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Abstract. The monograph contains a systematic treatment of a circle of 
problems in analysis and integral geometry related to inversion of the Radon 
transform on the space of real rectangular matrices. This transform assigns to 
a function / on the matrix space the integrals of / over the so-called matrix 
planes, the linear manifolds determined by the corresponding matrix equations. 
Different inversion methods are discussed. They rely on close connection be- 
tween the Radon transform, the Fourier transform, the Garding-Gindikin frac- 
tional integrals, and matrix modifications of the Riesz potentials. A special 
emphasis is made on new higher rank phenomena, in particular, on possibly 
minimal conditions under which the Radon transform is well defined and can 
be explicitly inverted. Apart of the space of Schwartz functions, we also em- 
ploy L p - spaces and the space of continuous functions. Many classical results 
for the Radon transform on R™ are generalized to the higher rank case. 
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Introduction 



0.1. Preface 

One of the basic problems of integral geometry reads as follows [Gel] . Let X be 
some space of points x, and let T be a certain family of manifolds rCl Consider 
the mapping f(x) — ► /(r) = J f which assigns to / its integrals over all r G T. 
This mapping is usually called the Radon transform of /. How do we recover f{x) 
from /(r)? This rather general problem goes back to Lorentz, Minkowski, Blaschke, 
Funk, Radon etc., and has numerous applications; see [Eh], [GGG], [GGV], [H], 
[Na], [Ru3], [RK], [SSW], and references therein. 

A typical example is the so-called fc-plane transform on W 1 which assigns to 
f(x), x G R™, a collection of integrals of/ over all /c-dimensional planes in R". This 
well known transformation gets a new flavor if we formally replace the dimension n 
by the product nm, n > to, and regard MJ 1 " 1 as the space ^ffl nm of n x m matrices 
x = (xij). Once we have accepted this point of view, then f(x) becomes a function 
of matrix argument and new "higher rank" phenomena come into play. 

Let Vn.n-k be the Stiefel manifold of orthonormal (n — fc)-frames in R™, i.e., 
V n , n -k = {£, ■ £ e ffl n , n - k , £,'£, = In-k} where £' denotes the transpose of £, J n _ fc 
is the identity matrix, and £'£ is understood in the sense of matrix multiplication. 
Given £ G V n ,n-k and t G 9K„_fc,m, we define the matrix k-plane 

(0.1) r = t(£, t) = {x : x G 97l„, m , £'x = t}. 

Let T be the manifold of all matrix fc-planes in 9Jl„. m . Each r G X is topologically 
an ordinary fcTO-dimcnsional plane in R nm , but the set T has measure zero in the 
manifold of all such planes. The relevant Radon transform has the form 



(0-2) f{r) = / /(a), r G X, 

the integration being performed against the corresponding canonical measure. If 
to = 1, then T is the manifold of fc-dimensional affine planes in W 1 , and /(r) is the 
usual fc-plane transform. The inversion problem for (0.2) reads as follows: How do 
we reconstruct a function / from the integrals /(r), r G X? Our aim is to obtain 
explicit inversion formulas for the Radon transform (0.2) for continuous and, more 
generally, locally integrable functions / subject to possibly minimal assumptions at 
infinity. 

A systematic study of Radon transforms on matrix spaces was initiated by 
E.E. Petrov [PI] in 1967 and continued in [C], [Grl], [P2]-[P4], [Shi], [Sh2]. 
Petrov [PI], [P2] considered the case k — n — m and obtained inversion formulas 
for functions belonging to the Schwartz space. His method is based on the classical 
idea of decomposition of the delta function in plane waves; cf. [GShl]. Results of 
Petrov were extended in part by LP. Shibasov [Sh2] to all 1 < k < n — m. 
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The following four inversion methods arc customarily used in the theory of 
Radon transforms. These are (a) the Fourier transform method (based on the 
projection-slice theorem), (b) the method of mean value operators, (c) the method of 
Riesz potentials, and (d) decomposition in plane waves. Of course, this classification 
is vague and other names are also attributed in the literature. One could also 
mention implementation of wavelet transforms, but this can be viewed as a certain 
regularization or computational procedure for divergent integrals arising in (b)-(d). 
In Section 0.2, we briefly recall some known facts for the fc-plane transform on W l 
which corresponds to the case to = 1 in (0.2). Our task is to extend this theory 
to the higher rank case to > 1. The main results are exhibited in Section 0.3. 
A considerable part of the monograph (Chapters 1-3) is devoted to developing a 
necessary background and analytic tools which one usually gets almost "for free" 
in the rank-one case. We are not concerned with such important topics as the 
range characterization, the relevant Palcy- Wiener theorems [P3], [P4], the support 
theorems, the convolution-backprojection method, and numerical implementation. 
One should also mention a series of papers devoted to the Radon transform on 
Grassmann manifolds; see, e.g., [Go], [GK1], [GK2], [Gr2], [GR], [Ka], [Ru5], 
[Str2], and references therein. This topic is conceptually close to ours. 



0.2. The fc-plane Radon transform on R™ 

Many authors contributed to this theory; see, e.g., [GGG], [H], [Ke], [Ru4] 
where one can find further references. We recall some main results which will be 
extended in the sequel to the higher rank case. 



0.2.1. Definitions and the framework of the inversion problem. Let 

Gn.k be the manifold of affine fc-dimensional planes in R n , < k < n. The integral 
(0.2) with to = 1 can be interpreted in different ways. Namely, if we define a k- 
plane r € G n ,k by (0.1) where £ € V n ,n-k and t G R n ~ k is a column vector, then 
the k -plane Radon transform is represented as 



(0.3) f{r) = f{i,t)= J f{y + &)dty, 

where d^y is the induced Lebesgue measure on the plane {y : £fy = 0}. For 
k = n — 1, this is the classical hyperplane Radon transform [H], [GGV]. Clearly, 
the parameterization t = r(£, t) is not one-to-one. An alternative parameteri- 
zation is as follows. Let r\ belong to the ordinary Grassmann manifold G n ^ of 
/c-dimensional linear subspaces of R™, and rj 1 - be the orthogonal complement of r\. 
The parameterization 

t = t(toA), neG nik , Aery- 1 , 

is one-to-one and gives 

(0.4) />) = />?, A) = [f(y + X)dy. 
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The corresponding dual Radon transform of a function ip(r) on Q n .k is defined 
as the mean value of y(r) over all fc-dimensional planes r through x £ M n : 

(0.5) ${%)= j filVo + x)dj = / ip(r],Pr n ±x)di]. 

SO(n) G„, fc 

Here, rjo is an arbitrary fixed fc-plane through the origin, and Pt v ±x denotes the 
orthogonal projection of x onto r\ . A duality relation 



(0.6) / f(x)<fi(x)dx = J f(T)<p(T)dT, 

R" Sn.k 

holds provided that either side of this equality is finite for / and ip replaced by |/| 
and \ip\, respectively [H], [So]. Here and on, dr stands for the canonical measure 
on Q n ,k\ see [Ru4]. 

The fc-plane transform is injective for all < k < n on "standard" function 
spaces where it is well defined (see Theorem 0.1 below). If k < n — 1, then the 
inversion problem for / is ovcrdetermincd because 

dim^fc = (n-k){k + l)>n = dimR", 

and the dual Radon transform is non-injective. If k = n — 1, then the dimension 
of the affine Grassmanian Gn,n-i is just n. In this case, the Radon transform and 
its dual are injective simultaneously; see also [Ru5] where this fact is extended to 
Radon transforms on affine Grassmann manifolds. 

The following theorem specifies standard classes of functions for which the fc- 
plane transform is well defined. 

Theorem 0.1. 
(i) Let f{x) be a continuous junction on R™ satisfying f(x) = 0(|x|~ a ). If a > k 
then the Radon transform /(r) is finite for all r £ Q n ,k- 

(ii) If (1 + \x\) a f (x) e L 1 for some a >k — n, in particular, if f G L p , 1 < p < n/k, 
then /(r) is finite for almost all r £ Q n ,k- 

Here, (i) is obvious, and (ii) is due to Solmon [So]. The conditions a > fc, 
a > k — n, and p < n/k cannot be improved. For instance, if p > n/k and 
f(x) = (2 + |x|)-"/P(log(2 + l^l))- 1 (e L' } ), then ,/(r) = oo. 

More informative statements can be found in [Ru4]. For instance, if |r| denotes 
the euclidean distance from the fc-plane r to the origin, then 

r ~ iri A ~™ r I I a— ri 

(0J) J fiT) (TTW)^ dT - C J ^IJTWW^^ 

9n,k K™ 

_ r((A-fc)/2)r(n/2) 

c ^r(A/2)r (( n-fc)/2)' ReX>k 

(see Theorem 2.3 in [Ru4]). Choosing A = n, we obtain 

(0 8) / hT) dr = f M dx 

( ' J (l + |r|2)«/2 aT J (l + \ X \2)(n-k)/2 aX - 

Gn,k R" 

It is instructive to compare (0.8) with [So, Theorem 3.8] where, instead of explicit 
equality (0.8), one has an inequality (which is of independent interest!). Different 
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modifications of (0.7) and (0.8), which control the Radon transform and its dual at 
the origin and near spherical surfaces, can be found in [Ru4]. 

It is worth noting that "Solmon's condition" (1 + \x\) k ~ n f{x) e i 1 (R") is not 
necessary for the a.e. existence of / (t) because, the Radon transform essentially has 
an exterior (or "right-sided") nature; see (0.11) below. For any rapidly decreasing 
function / £ C°°(IR™\{0}) satisfying f(x) ~ \x\~ n as \x\ — * 0, the Solmon condition 
fails, but /(£,£) < oo a.e. (it is finite for all r ^ 0). However, if for / > 0, 
one also assumes /(t) e Lj oc (in addition to /(r) < oo a.e.), then, necessarily, 
(1 + \x\) k ~ n f{x) E L 1 [So, Theorem 3.9]. 

0.2.2. The fc-plane transform and the dual fc-plane transform of radial 
functions. There is an important connection between the transformations / — ► / 
and tp — > <p, and the Riemann-Liouville (or Abel) fractional integrals 



(0.9) (/«/)(*) = — ly-^dr, { r ms)= -L^-L-dr, 



T(a)J {s-r) L a "(a)y (r-s) s 

s 

Re a > 0. Let cr„_i = 2W 2 /T(n/2) be the area of the unit sphere S^ 1 in E™. 
The following statement is known; see, e.g., [Ru4]. 

Theorem 0.2. For x e W 1 and r = (ry, A) e Q n ,k, let 

(0.10) r = \x\ =dist(o, x), s = |A| = dist(o,r) 

denote the corresponding euclidean distances from the origin. If f(x) and <p{r) 
are radial functions, i.e., f(x) = fo(r) and (p(r) = ifo(s), then /(r) and <p(x) are 
represented by the Abel type integrals 

DC 

(0.11) />) = a k -i [ h(r){r 2 - s^-'rdr, 



(0.12) ^(x) = gfc - 1<T "^- 1 / Ms)(r 2 ^ 2 - 1 s n - k - 1 ds, 

0~n—l T J 



provided that these integrals exist in the Lebesgue sense. 

Obvious transformations reduce (0.11) and (0.12) to the corresponding frac- 
tional integrals (0.9). 

0.2.3. Inversion of the fc-plane transform. Abel type representations (0.11) 
and (0.12) enable us to invert the fc-plane transform and its dual for radial func- 
tions. To this end, we utilize diverse methods of fractional calculus described, e.g., 
in [Rul] and [SKM]. In the general case, the following approaches can be applied. 

The method of mean value operators. The idea of the method amounts to 
original papers by Funk and Radon and employs the fact that the fc-plane transform 
commutes with isometrics of W 1 . Thanks to this property, the inversion problem 
reduces to the case of radial functions if one applies to /(r) the so-called shifted 
dual k-plane transform (this terminology is due to F. Rouvicre [Rou]). The latter 
is defined on functions </?(r), r e G n ,ki by the formula 

(0.13) <p r {x) = J lfi(jT r + x)d-f, 

SO(n) 
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where r r is an arbitrary fixed /c-plane at distance r from the origin. The integral 
(0.13) is the mean value of ip(r) over all r at distance r from x. For r = 0, it 
coincides with the dual /c-plane transform y>(x). Modifications of (0.13) for Radon 
transforms on the n-dimensional unit sphere and the real hyperbolic space were 
introduced by Helgason; see [H] and references therein. 
The core of the method is the following. 

Lemma 0.3. Let <p = f, f e L' p , 1 < p < n/k, and let 
(0.14) (X r /-)( x ) = _L f f( x + r e)d9 1 r>0, 

O n -l J 

s „-i 

be the spherical mean of f. If g x (s) = (-M^f)(x) and ip x (s) = ir~ k > 2 (f>r^(x), then 

(0.15) {I- /2 9x){s)=M')- 

If k is even, (0.15) yields 



(0.16) m ^-^.-- } Mx) ^ 

where, for non-smooth /, the equality is understood in the almost everywhere sense. 
This inversion formula is of local nature because it assumes that ip r (%) IS known 
only for arbitrary small r. The case of odd k is more delicate. If / is continuous 
and decays sufficiently fast at infinity, then (0.15) is inverted by the formula 

(0.17) f(x)=(-^-) m (r- k/2 i, x )(s) , VmeN, m>fc/2, 

V ds/ s=o 

which is essentially non-local. For / belonging to L p , the integral (I m 1 Px){s) 
can be divergent if n/2m < p < n/k. It means that the inversion method should 
not increase the order of the fractional integral. This obstacle can be circumvented 
by making use of Marchaud's fractional derivatives; see [Ru4, Section 5] for details. 
The method of Riesz potentials and the method of plane waves. It is instructive 
to describe these methods simultaneously because both employ the Riesz potentials 

(0.18) (i°f)( x ) = -}— [f(x-y)\y\ a - n dy, iel°, 

(0.19) ln ( a )= LJ; Rea>0, a^n,n + 2,.... 

r((n-a)/2) 

The Fourier transform of the corresponding Riesz distribution \x\ a ~ n /"/ n (a) is \y\~ a 
in a suitable sense. It means that I a can be regarded as (— a/2)th power of —A, 
where A = d 2 /dx\ + • • • + d 2 /dx 2 l is the Laplace operator . For / e L p (R n ), the 
integral (0.18) absolutely converges if and only if 1 < p < n/Re a. For Re a < and 
sufficiently smooth /, the Riesz potential /"/ is defined by analytic continuation so 
that / / = /. Numerous inversion formulas for I a f are known in diverse function 
spaces, sec [Rul], [Ru4], [Sa], [SKM], and references therein. 

The method of plane waves is based on decomposition of the Riesz distribution 
in plane waves; see [GShl] for the case k = n— 1. Technically, this method realizes 
as follows. In order to recover / at a point x from the data /(t) = /(£, t), one first 
applies the Riesz potential operator of the negative order — k (this is a differential or 



/ HtWt 


r\ a+k - n dx, 
- r | a+fe -™ dr. 


(~\ / J\ x )\ x 
1 f -(r)\x 


ln- k (a) J A ' )IX 

Qn.k 
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pseudo-differential operator) in the t-variable, and then integrates over all fc-plancs 
through x, i.e., applies the dual Radon transform. This method requires / to be 
smooth enough. Another method, referred to as the method of Riesz potentials, is 
applicable to "rough" functions as well. Now we utilize the same operators but in 
the reverse order: first we apply the dual Radon transform to /(r), and then the 
Riesz potential operator of order — k in the x-variable. This method allows us to 
reconstruct any function / G L p , 1 < p < n/k, for almost all x G M. n [Ru4]. 

Both methods can be realized in the framework of analytic families of inter- 
twining fractional integrals 

(0.20) (P a f)(r) 



(0.21) (P»(aO 



i?ea>0, a + k -n ^ 0,2,4, . .. , 

where |x — t denotes the euclidian distance between the point x and the fc-plane r. 
These operators were introduced by Semyanistyi l [Se] for k = n — 1. They were 
generalized in [Ru2] and subsequent publications of B. Rubin to all < k < n and 
totally geodesic Radon transform on spaces of constant curvature; see also [Ru4]. 
Semyanistyi's fractional integrals have a deep philosophical meaning. One can write 

(0.22) p a f^I a f j p a i P = (I a i P ) v , 

where for f{r) = y>(£, t), I a ip denotes the Riesz potential on M. n ~ k in the t- variable. 
If / and ip are smooth enough, then the equalities (0.22) extend definitions (0.20) 
and (0.21) to all complex a =/= n — k, n — k + 2, . . . . In particular, for a = 0, we get 
the /c-plane transform and the dual fc-plane transform, respectively. 

The following statement generalizes the well known formula of Fuglcdc 

(0.23) (/) v = c k , n I k f, c k , n = (2 7 r) fe a„_ fc _ 1 /a„_ 1 ; 

see [Fu], [So, Theorem 3.6], [H, p. 29]. 

Theorem 0.4. LetfeLP, 1 <p < n/(Rea+ fc), Rea>0. Then 

(0.24) P a f = c k , n I a+k f. 

The equality (0.24) gives a family of inversion formulas (at least on a formal 
level) : 

(0.25) c fc ,„/ = r a - k P "/. 

In the case a = 0, the last equality reads 

(0.26) c fc>n / = D fe <^, v = f. 

Here, D* = I~ k = (— A) k / 2 denotes the Riesz fractional derivative that can be 
realized in different ways depending on a class of functions / and the evenness of 



Vladimir IFich Semyanistyi (1925-1984), a talented mathematician whose papers have 
played an important role in integral geometry and related areas of analysis. 
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k. For example, if / is good enough, the fc-plane transform / = ip can be inverted 
by repeated application of the operator —A. Namely, for k even, 

(0.27) c k , n f(x) = (-A) fe / 2 ^(x). 

If k is odd and 1 < k < n — 2, then 

(0.28) c k , n f(x) = (-A)( k +V/ 2 (I l p)(x). 

Inversion formulas (0.26)-(0.28) are typical for the method of Riesz potentials. 
Furthermore, if a = —k, then (0.25) gives 

(0.29) c fe ,„/=p-V <p(£,t) = fa,t). 

The last formula can be regarded as a decomposition of / in plane waves. Different 

realizations of operators D fc and P ~ k can be found in [Rul] and [Ru4]; see also 
[Ru6]. 

0.3. Main results 

This section is organized so that the reader could compare our new results with 
those in the rank-one case. That was the reason why subsections below are entitled 
as the similar ones in Section 0.2. The general organization of the material is clear 
from the Contents. 

0.3.1. Definitions and the framework of the inversion problem. In 

order to give precise meaning to the integral (0.2), we use the parameterization 
t = t(£, t), £ G V nin -k, t G 9Tt„_fe jTO , of the matrix fc-plane (0.1) and arrive at the 
formula 

(0.30) /(£,*) = / f(V + &)dtv, 

where d^y is the induced measure on the subspace {y : y G $H„. m ; £'y — 0}. This 
agrees with the formula (0.3) for the fc-plane transform and can be also written in 
a different way, see (4.10). Another parameterization 

t = t(t],X), r/eG nik , X = [Ai . .. A m ] G 9K„, m , A, G r/- 1 , 

which is one-to-one (unlike that in (0.30)), gives 



(0.31) /(r) = dyi... f([yi +Xi...y m + X m }) dy m , 

n v 

cf. (0.4). 

We first specify the set of all triples (n, m, k) for which the Radon transform is 
injective. It is natural to assume that the transformed function should depend on 
at least as many variables as the original one, i.e., 

dimT > dim$H„ im . 

By taking into account that dimV^ iTO = m(2n — m — l)/2 [Mu, p. 67], we have 

dimT = dim(V^, n _fc x 9Jl„_fe iTO )/0(n - k) 

= {n-k)(n + k- l)/2 + m(n - k) - (n - k){n -k- l)/2 

= (n — k)(k + m). 
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The inequality (n — k) (k + m) > nm implies the natural framework of the inversion 
problem, namely, 

(0.32) 1 <k <n-m, m > 1. 

We show (Theorem 4.9) that the Radon transform / — » / is injective on the 
Schwartz space S(9Jl„ >m ) if and only if the triple (n,m,k) satisfies (0.32). The 
proof of this statement relies on the matrix generalization of the projection- slice 
theorem (Theorem 4.7) which establishes connection between the matrix Fourier 
transform and the Radon transform. For the case m — 1, we refer to [Na, p. 11] 
(k = n - 1) and [Ke, p. 283] (any < k < n); see also [Shi], [Sh2], and [OR] for 
the higher rank case. 

We also specify classes of functions for which the Radon transform is well 
defined. Let I m be the identity m x m matrix and let \a\ denote the absolute 
value of the determinant of a square matrix a. The following statement agrees with 
Theorem 0.1. 

Theorem 0.5. 
(i) Let f(x) be a continuous function on 9Jt„ ]m , satisfying f(x) — 0(\I m +x' x\~ a ' 2 ) . 
If a > k + m — 1, then the Radon transform f(r) is finite for all r G X. 
(ii) If \I m + x'x\ a ' 2 f(x) G i 1 ($H„. m ) for some a > k — n, in particular, if f G 
L p (dJl nim ), 1 < p < po, Po = (n + m— l)/(k + m— 1), then /(r) is finite for almost 
all t el. 

As in Theorem 0.1, the conditions a > k + m — 1, a > k — n, and p < po cannot 
be improved. For instance, if p > p and f(x) = \2I m + x / x\- ( - n+m - 1 ^ 2p (log \2I m + 
x'x])- 1 (G U>(m n , m )), then />) = oo. 

In fact, we have proved the following generalization of (0.7) which implies (ii) 
above: 

1 /• r k'£|( A -™)/ 2 



(0.33) 



, x , x \{\-n)/2 



= C J f{x) \I m + x> X \^-W dX > R^>k + m-l, 

where the constant c is explicitly evaluated; see (4.48), (4.42). 

We introduce the dual Radon transform which assigns to a function (p(r) on X 
the integral over all matrix fc-planes through x G 97l„ !m : 

(0.34) #3) - J <p(r). 

tBx 

In terms of the parameterization r = T(^,t), £ G V n _ n -k, t G 9Jt n _fc jTO , this integral 
reads as follows: 

(0.35) <p(x) = —!— f <p(S,?x)dt, 

where u n ^ n -k is the volume of the Stiefel manifold V n , n -k', see (1.37). The integral 
(0.35) is well defined for any locally integrable function ip. We do not study the 
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inversion problem for the dual Radon transform here. However, it is natural to 
conjecture that the dual Radon transform is injective if and only if k > n — m, 
and the relevant inversion formula can be derived from that for the Radon trans- 
form. The respective results were obtained in [Ru5] for Radon transforms on affinc 
Grassmann manifolds. We plan to study this topic in forthcoming publications. 

0.3.2. The Radon transform and the dual Radon transform of radial 
functions. We call functions / on Wl n , m and if on 1 radial if they are 0(n) left- 
invariant. Each radial function f(x) on 9Jt n . m (</?(£,£) on X, resp.) has the form 
f(x) — fo(x'x) (</?(£>£) — Vo(t't), resp.). We establish connection between the 
Radon transforms of radial functions and the Gdrding- Gindikin fractional integrals 

s 

(0.36) (/«/)(*) = -L- f(r)\s-r\ a - d dr, 

r TO (a) J 

o 

oo 

(0.37) {I1f){s) = -L-, [f(r)\r-sr d dr 

r TO (a) J 

s 

associated to the cone V m of positive definite symmetric m x m matrices. Here, 
d = (m + l)/2 and T m (a) is the generalized gamma function (1.6) which is also 
known as the Siegel gamma function [Si]. In (0.36), s belongs to V m , and integration 
is performed over the "interval" (0, s) — {r : r 6 V m , s — r e V m }. In (0.37), s 
belongs to the closure V m , and one integrates over the shifted cone s + ? m = {r : 
r E V m , r — s E Vm}- For m = 1, the integrals (0.36) and (0.37) coincide with the 
Riemann-Liouville fractional integrals (0.9). Section 2.1 contains historical notes 
related to integrals I±f in the higher rank case. 

For sufficiently good /, the integrals I±f converge absolutely if Re a > d — 1 
and extend to all a € C as entire functions of a. If a is real and belongs to the 
Wallach-like set 

( 1 3 m— ll I „ m — 

w m = |0,-,1,-,...,— 2~ Juja: Rea> — 

then I±f are convolutions with positive measures; see (2.12), (2.22). The converse 
is also true; cf. [FK, p. 137]. Owing to this property, if a £ W m , then one can 
define the Garding-Gindikin fractional integrals for all locally intcgrablc functions 
which in the case of I" obey some extra conditions at infinity. Different explicit 
formulas for I±f, a G W m , are obtained in Section 2.2. 

A natural analog of Theorem 0.2 for the higher rank case employs fractional 
integrals (0.36)-(0.37) and reads as follows. 

Theorem 0.6. For x £ 9Jt„, TO and t e Wl n -k, m , let r = x'x, s = t't. 
(i) If f{ x ) is radial, i.e., f(x) = fo(x'x), then 

(0.38) f(U)=ir km/2 (I- /2 f°)(s)- 

(ii) Let <p(t) be radial, i.e., ip(£,t) = ipo(t't). We denote 

$o(s) = \s\ 5 (f (s), 6 = (n-k)/2-d, d=(m + l)/2. 
If 1 < k < n — m, then for any matrix x e ^XH n ,m of rank m, 

(0.39) ^(x)= C |r| d -"/ 2 (4 /2 <D )(r), c - K km ' 2 o- n _ k , m /a„ , m , 
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o~ n ,m being the volume of the Stiefel manifold V n;m . Equalities (0.38) and (0.39) 
hold for any locally integrable functions provided that either side exists in the 
Lebesgue sense. 

0.3.3. Inversion of the Radon transform. Theorem 0.6 reduces inversion 
of the Radon transform and the dual Radon transform of radial functions to the 
similar problem for the Garding-Gindikin fractional integrals. If / belongs to the 
space T)(Vm) of infinitely diffcrcntiable functions compactly supported away from 
the boundary dV m , we proceed as follows. For r = {r^j) G V m , we consider the 
differential operators D± defined by 



3 \ ( 1 if % = j 



D + = det ( rnj ^-) , nu = { 1/2 a l+ J , D. = i-lTD 



+ ■ 



If a G C and j G N, then 

Dii^f = iiDif = ir j f, f e v{v m ). 

Hence I±f can be inverted by the formula 

(0.40) f = Dilt a g, g = I}f, 

where j > a if a = 1/2, 1, . . . , and j > Re a + d — 1 otherwise. For "rough" 
functions /, we interpret (0.40) in the sense of distributions; see Lemmas 2.23 and 
2.24. A challenging open problem is to obtain "pointwisc" inversion formulas for 
the Garding-Gindikin fractional integrals similar to those for the classical Ricmann- 
Liouvillc integrals [Rul] , [SKM] . It is desirable these formulas would not contain 
neither operations in the sense of distributions nor the Laplace (or the Fourier) 
transform, i.e., the problem would be resolved in the same language as it has been 
stated. 

In the general case, we develop the following inversion methods for the Radon 
transform which extend the classical ones to the higher rank case. 

The method of mean value operators. The Radon transform /(t) commutes 
with the group M(n,m) of matrix motions sending x € 9K n ,m to r )xf3 + b, where 
7 G O(n) and (3 G 0(m) are orthogonal matrices, and b G 9Jl„. m . This property 
allows us to apply a certain mean value operator to /(r) and thus reduce the 
inversion problem to the case of radial functions. 

Let us make the following definitions. We supply the matrix space 2H n . m and 
the manifold X of matrix fc-planes with an entity that serves as a substitute for 
the euclidean distance. Specifically, given two points x and y in 9Jl„. m , the matrix 
distance d(x, y) is defined by 

d(x,y) = [(x- y y(x-y)}^. 

Accordingly, the "distance" between x G 5H„ jm and a matrix fc-plane r = r(£, t) G X 
is defined by 

d(x,T) = Wx-t)'^X-t)Yl 2 . 

Unlike the rank-one case, these quantities are not scalar-valued and represented 
by positive semi-definite matrices. Radial functions f(x) on 9Jt„. m and ip(r) on X 
virtually depend on the matrix distance to the 0-matrix from x and r, respectively. 
For functions </?(t), t G X, we define the shifted dual Radon transform 

(0.41) M*)= [ ^( T )> sgTV 

Jd(x,T)=S 1 / 2 
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This is the mean value of (p(r) over all matrix planes at distance s 1 ' 2 from x. For 
s = 0, it coincides with the dual Radon transform <p(x). A matrix generalization 
of the classical spherical mean is 

(M r /)(x) = — !- f f(x + vr^dv, r G V m . 

0~n,m J 

V n , m 

The following statement generalizes Lemma 0.3 to the case m > 1. 

Lemma 0.7. For fixed x E 9Jt n , m , let F x (r) = (M r f){x). If f e L p (9tt n , m ), 
l<p<(n + m— l)/(fc + m — 1), then 

(0.42) (fy s (x)=ir km / 2 (I k _ /2 F x )(s), seV m . 

Owing to (0.42) and the inversion formula for the Garding-Gindikin fractional 
integrals, one can recover f(x) from the Radon transform f(r) as follows. 

Theorem 0.8. Let f e L p (2tt„, m ), 

n + to — 1 



1 < k < n — m, 1<J5< 



TO — 1 

-(/>( T ) = /( T ); ^en 

(0.43) f(x) = ^- fo "/ 2 lhn(D! /2 $ x )(r), $,(5) = 0,(x), 

where D_ = I_ is understood in the sense of T>' -distributions. 

The method of Riesz potentials and the method of plane waves. We extend the 
method of Riesz potentials to the higher rank case and give a new account of the 
method of plane waves. The second method was developed by E.E. Petrov [PI] for 
the case k = n — to, and outlined by L.P. Shibasov [Sh2] for all 1 < k < n — to. 
Both methods employ the following matrix analog of the Riesz potential (0.18): 

(0.44) (I a f)(x) = ^ I f(*-y)\y\™ n dy, flea > to -1, 

ln,m\a) J 

VR„,m 

where \y\ m = det(y'y) 1/2 , 

(0.45) 7„ (a) = — — — ; ^— - — , a^n-TO + 1, n-TO + 2, .... 

r m ((n-a)/2) 

We consider the Cayley- Laplace operator 

(A/)(X) - (det(d , d)f)(x), X = (Xij) £ Wn,m, 

where d is the n x m matrix whose entries are partial derivatives d/dxij. The 
name "Cayley-Laplace" was attributed to this operator by S.P. Khekalo [Kh] and 
suits it admirably. On the Fourier transform side, the action of A represents a 
multiplication by (— l) m |j/|^. Since the Fourier transform of the Riesz distribution 
\ x \rn n Hn,m{o) is |y|~" in a suitable sense, then I a can be regarded as (— a/2)th 
power of the operator (— l) m A. For Re a < m — 1, a =/= n — to + 1, n — to + 2, . . . 
and sufficiently smooth /, the Riesz potential is defined as analytic continuation of 
the integral (0.44). If a is real and belongs to the corresponding Wallach-like set 

W„. m = {0, 1,2, . . . , fco} U {a : Rea>m—1; a jt n — m + 1, n — m + 2, . . .}, 
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fco = min(m — 1, n — m), 
then I a f is a convolution with a positive measure; see (3.54)-(3.56). This result 
allows us to extend the original definition of the Riesz potential to locally integrable 
functions provided a G W„ iTO . It was shown in [Ru7], that if 

Ti 

(0.46) fEL p (m n , m ), 1<P<13 ; 7, aeW„. m , 

Re a + to — 1 

then (I a f)(x) is finite for almost all x G Wl n , m . On the other hand, if 

n + to — 1 

then there is a function / G L p (dJl n ^ m ) such that (I f)(x) = oo (see (6.16), Theorem 
4.27, and Appendix B). It is still not clear what happens if 

ir, ,„s n - n + m — 1 

(°' 47 ) p 1 T - p< p 1 T • 

Re a + to — 1 rte a + to — 1 

This gap represents an open problem. 

As in the rank-one case, the method of Riesz potentials and the method of 

plane waves employ intertwining operators 

(0.48) P a f = I a f, P<V=(/» V , 

qeC, a ^ n — k — TO + 1, n — k — to + 2, ... . 
Here, 1 < fc < n — to and I a is the Riesz potential on %R n _k,m acting in the 
t- variable. If 

Re a > m — 1, a ^ n — k — to + 1, n — k — to + 2, ..., 

then 

(0.49) (P a m,t) = 1 —- f f{x)Wx-t\^ k - n dx, 

9JL, m 

(0.50) (P»(s) = l —- [d.S [ <p(U)\?x-t\£- k - n dt, 

v»,»-» 2K n _ fe , m 

where d*£ = cr~ n _ k d£ is the normalized measure on V n , n -k and 7n-fc, m (a) is the 
normalizing constant in the definition of the Riesz potential on 9Jl„_ / t. m ; cf. (0.45). 

We call P a f and P a tp the generalized Semyanistyi fractional integrals. For a = 0, 
operators (0.48) include the Radon transform and its dual, respectively. In the case 
to = 1, the integrals (0.49) and (0.50) coincide with those in (0.20) and (0.21). 

Theorem 0.9. Let 1 < k <n — m, a G W„ !m . Suppose that 



f G L p , 1 < p < 



ii 



Then 




(0.51) 


(P a f)(x) = c n , k , m (I a+k f)(x), 




r,km_km/2- n ( n \ /T , (n — k 


In particular, 




(0.52) 


{ff{x) = c n , k , m {I k f){x) 
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(the generalized Fuglede formula). 

Theorem 0.9 is a higher rank copy of Theorem 0.4. Thus, as in the case m = 1, 
inversion of the Radon transform reduces to inversion of the Riesz potential. Unlike 
the rank-one case, we have not succeeded to obtain a pointwisc inversion formula 
for the higher rank Riesz potential so far, and use the theory of distributions. The 
properly defined space $(37t n , m ) of test functions is invariant under the action of 
Riesz potentials, and 

(0.53) {I- a 4>){x) = {F- 1 \y\% l ?4>){x), <\> e $, aeC, 

in terms of the Fourier transform. 

Theorem 0.10. Let f e L p (Wl nim ), 1 < p < n/(k + m — 1), so that the Radon 
transform <p(t) = /(t) is finite for almost all matrix k-planes r. The function f 
can be recovered from ip in the sense of $' -distributions by the formula 

(0.54) c„, fc , m (/,0) = (^,/- fe 0), 0G$, 

where I~ k is the operator (0.53). In particular, for k even, 

(0.55) Cn, k , m (f, <t>) = (-l) mfe / 2 (^, A fc / 2 0), 4> e $. 

This theorem reflects the essence of the method of Riesz potentials. The case 
a = —k in (0.51) provides the following inversion result in the framework of the 
method of plane waves. 

Theorem 0.11. Let 1 < k < n — m. If f belongs to the Schwartz space 
S(2ftn,m); then the Radon transform ip = f can be inverted by the formula 

(0.56) c nikim f(x) = {p- k <p)(x). 

Explicit expressions for P~ k ip are given by (6.21)-(6.23). 
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CHAPTER 1 



Preliminaries 



In this chapter, we establish our notation and recall some basic facts important 
for the sequel. The main references are [FZ], [Herz], [Mu], [T]. 

1.1. Matrix spaces. Notation 

Let yJt n , m be the space of real matrices having n rows and m columns; 9Jt^ ' m C 
9ftn,m is the subset of all matrices of rank k. We identify 9Jt„. m with the real Eu- 
clidean space M. nm . The latin letters x, y, r, s, etc. stand for both the matrices 
and the points since it is always clear from the context which is meant. We use a 
standard notation O(n) and SO(n) for the group of real orthogonal n x n matri- 
ces and its connected component of the identity, respectively. The corresponding 
invariant measures on 0(n) and SO(n) are normalized to be of total mass 1. We 
denote by M(n,m) the group of transformations sending x G 9Jl n .m to jxf3 + b, 
where 7 G O(n), f3 G 0(m), and b G 9Jt„. m . We call M{n 1 m) the group of matrix 
motions. 

If x = (xij) G fffl n ,m, wc write dx = Y[i=i IlfLi ^ x i,j f° r the elementary volume 
in VJl n , m . In the following, x' denotes the transpose of x, I m is the identity mx m 
matrix, and stands for zero entries. Given a square matrix a, we denote by det(a) 
the determinant of a, and by \a\ the absolute value of det(a); tr(a) stands for the 
trace of a. 

Let S m be the space ofmxm real symmetric matrices s — (s»,j), Sjj = Sj^. It is 
a measure space isomorphic to R m ( m+1 )/ 2 with the volume element ds — Y\i<j dsij. 
We denote by V m the cone of positive definite matrices in S m ; V m is the closure 
of V m , that is the set of all positive semi-definite m x m matrices. If m = 1, then 
V m is the open half-line (0, 00) and V m = [0, 00). For r G V m {r G V m ) we write 
r > (r > 0). Given s\ and S2 in S m , the inequality s\ > S2 means Si — S2 G V m - 
If a G V m and b G V m , then the symbol J f(s)ds denotes the integral over the set 

{s : s G V m , a < s < b) — {s : s — a G V m , b — s G V m }- 

For s G S m , wc denote by s+ the function which equals \s\ x for s G V m and zero 
otherwise; s^ equals (— s)+. 

The group G = GL(m, R) of real non-singular mxm matrices g acts transitively 
on V m by the rule r — > g'rg. The corresponding G-invariant measure on V m is 

(1.1) d*r = \r\- d dr, \r\ = det(r), d=(m + l)/2 

[T, p. 18]. The cone V m is a G-orbit in S m of the identity matrix I m - The boundary 
dV m of "P m is a union of G-orbits of m x m matrices 

«»-[? J], *-0,l,...,m-l. 

15 
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More information about the boundary structure of V m can be found in [FK, p. 72] 
and [Bar, p. 78]. 

Let T m be the subgroup of GL{m 1 R) consisting of upper triangular matrices 



(1.2) 



t = 



E-1,1 



I'* 



U.i > o, 



i>m(m— 1)/2 



t* = {Uj : i < j] e 

Each r G V m has a unique representation r = t't, t G T m , so that 



/(r-)dr 



oo oo 



(1.3) 



./(*i,i,- 



■ ; ^m.m) &t'm,mi 



f(ti,i,...,t m ,m) = 2 m / f{t't)dU, dU=Y[dt tJ , 

R m(m-l)/2 «<J 

[T, p. 22] , [Mu, p. 592]. In the last integration, the diagonal entries of the matrix 
t are given by the arguments of /, and the strictly upper triangular entries of t are 
variables of integration. 

We recall some useful formulas for Jacobians; see, e.g., [Mu, pp. 57-59]. 

Lemma 1.1. 
(i) If x = ayb where y G $H„ jm , a G GL(n, M), and b G GL(to,IR), then dx — 
\a\ m \b\ n dy. 



(ii) If r — q' sq where s G S m , and q G GL(m,W), then dr 
(iii) If r = s where s G V m , then r G V m , and dr — |s|~ 



\<1 



ro+l 

l ds. 



ds . 



Some more notation are in order: N denotes the set of all positive integers; 
R+ = (0, oo); Sij is the usual Kronecker delta. The space C(jM n ^ m ) of continuous 
functions, the Lebesgue space L p (9Jt„ im ), and the Schwartz space S(Wl n . m ) of C°° 
rapidly decreasing functions are identified with the corresponding spaces on R" m . 
We denote by C c (9Jl„. m ) the space of compactly supported continuous functions 
on Wl n ,m- The spaces C^iVm), C k (V m ) and S(V m ) consist of restrictions onto 
Vm of the corresponding functions on S m ~ M. N , N — m(m + l)/2. We denote 
by 1)(Vm) the space of functions / G C°°(7 : ' m ) with supp/ C V m ; Lj oc (P m ) is the 
space of locally integrable functions on V m . The Fourier transform of a function 
/ G L^^Slnm) is defined by 

(1.4) {ff){y)= I cxp(tr(iy'x))f(x)dx, y£Wl n , m . 

97L, m 

This is the usual Fourier transform on R nm so that the relevant Parseval formula 
reads 

(1.5) 



(.F/,.^) = (27rr m (/,^), 
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where 

(/, ip)= / f(x)ip(x) dx. 

9Jl n , m 
We write c, ci, C2, ■ • • for different constants the meaning of which is clear from the 
context. 

1.2. Gamma and beta functions of the cone of positive definite matrices 

The generalized gamma function associated to the cone V m of positive definite 
symmetric matrices is defined by 

(1.6) r m (a) = f cxp(-tr(r))\r\ a - d dr, d=(m+l)/2. 

v m 

This is also known as the Siegel integral [Si] (1935) arising in number theory. Actu- 
ally it came up earlier in statistics. Substantial generalizations of (1.6) and historical 
comments can be found in [Gil], [FK, Chapter VII], [T, p. 41], [GrRi, p. 800]. 
Using (1.3), it is easy to check [Mu, p. 62] that the integral (1.6) converges 
absolutely if and only if Re a > d — 1. Moreover, T m (a) is a product of ordinary 
gamma functions: 

m—l 

(1.7) T m (a) - ^(™-D/ 4 H T(a - j/2) . 

3=0 

This implies a number of useful formulas: 

(1-8) r ^ d - a ^ = ( -ir-£^ d=(m + l)/2, 

l m {d-a) I m (a-1) 

(1.9) r ro (l-a/2) = T(a + rn) = 

V ' K ' r m (-a/2) r(a) l ' 7 ' 

where (a, m) — a (a + 1) • • • (a + m — 1) is the Pochhammcr symbol. Furthermore, 
for 1 < k < m, k G N, the equality (1.7) yields 

(1 1Q) r m (a) = T k (a+(k-m)/2) 

( ' ' T m (a + k/2) " T k (a + k/2) ' 

(1.11) r ro (a) = ir k ( m ^/ 2 r k (a)T m _ k (a - k/2). 
In particular, for k = m — 1, 

(1.12) r m (a) - rr( m - 1 )/ 2 r m _ 1 ( a )r(a - (m - l)/2). 
The 6eia function of the cone ? m is defined by 

-?,„ 

(1.13) Bm(a,/3) = / \r\ a - d \I m - rf- d dr, 



where, as above, (i = (to + l)/2. This integral converges absolutely if and only if 
Re a, Re (3 > d — 1. The following classical relation holds 

/, 1/n o / m r m (q)r m (/3) 

(1-14 B m (a,/3) = 

T m (a + fi) 
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[FK, p. 130]. In Appendix A, we present a table of some integrals that can be 
easily evaluated in terms of the generalized gamma and beta functions. Most of 
these integrals are known but scattered all over the different sources. This table will 
be repeatedly referred to in the sequel. For the sake of completeness, we mention 
the paper [Ner] containing many other formulas of this type in the very general 
set-up. 

1.3. The Laplace transform 



in- 



Let T m — V m + i<Sm be the generalized half- plane in the space S^ = S m + iS, 
the complcxification of S m . The domain T m consists of all complex symmetric 
matrices z = a + iu; such that a = Re z G V m , and u) = Im z G S m . Let / be 
a locally integrable function on S m , f(r) — if r £ V m , and exp(— tr(<ror))/(r) G 
L l (S m ) for some <To G V m . Then the integral 



(1.15) (Lf)(z) = J exp(-tr(*r))/(r)dr 

V,n 

is absolutely convergent in the (generalized) half-plane Re z > uq and represents a 
complex analytic function there. Lf is called the Laplace transform of /. If 



(1.16) (J-g){uj) = I exp(tr(iws))g(s)ds, u> G S m , 

s,„. 
is the Fourier transform of a function g on S m , then 

(Lf)(z) = {?g*)(-w), 

where ga{r) — exp(— tr(or))/(r) G L l {S m ) for a > ctq. Thus, all properties of the 
Laplace transform are obtained from the general Fourier transform theory for Eu- 
clidean spaces [Herz], [VI, p. 126]. In particular, we have the following statement. 

Theorem 1.2. Let exp(— tr(o- r))/(r) G L 1 (7 , m ), <7 G V m , and let 

|(L/)((T + iuj)\doj < oo 



s m 
for some a > gq. Then the Cauchy inversion formula holds: 

^ / exp(tr(sz)(Lf)(z)dz = < 



(L17) (2^F / Q M^(sz)(Lf)(z)dz 



N = m{m + l)/2. The integration in (1.17) is performed over all z = a + ioj with 
fixed a > (To an d ^ ranging over S m . 

The following uniqueness result for the Laplace transform immediately follows 
from injectivity of the Fourier transform of tempered distributions. 

Lemma 1.3. If /i(r) and f 2 (r) satisfy 

(1.18) cxp(-tr( ( ror))/ J (r)GL 1 (7' m ), j = 1,2, 

for some o-q G V m , and (L/i)(z) = (Lf2)(z) whenever Re z > (To, then f\{r) = 
f 2 {r) almost everywhere on S m . 
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The convolution theorem for the Laplace transform reads as follows: If f\ and 
fi obey (1.18), and 

r 

(fi*f2)(r) = Jfi(r-a)f 2 (a)da 

o 
is the Laplace convolution, then 

(1.19) L(h * f 2 ){z) = {Lfi){z){Lf 2 )(z), Re z > a . 
Lemma 1.4. Let z eT m = V m + iS m , d = (to 4- l)/2. Then 

(1.20) /exp(-, r (, r ))| rP -* = r m ( )de,(,)-", «.„>„- 1. 

Here and throughout the paper, we set det(z)~" = exp(— alogdet(z)) where 
the branch of logdct(z) is chosen so that det(z) = \a\ for real z = cr G "P m . 

The equality (1.20) is well known, see, e.g., [Herz, p. 479]. For real z = a G 
Vrm it can be easily obtained by changing variable r — ► z~ 1 '' 2 rz~ 1 '' 2 . For complex 
z, it follows by analytic continuation. 

1.4. Differential operators D± 

Let r = (rij) G "P m . We define the following differential operators acting in 
the r- variable: 

(1.21) D + = D + , r = det [r, itj J->) , ViJ = | J /2 J J = J. 

(1.22) L>_ = D_, r = (-l)" 1 ^,,.. 

Lemma 1.5. /// G ©("Pm), on<i f/ie derivatives D^g exist in a neighborhood of 
the support of f , then 

(1.23) J (D ± f)(r)g(r)dr = J f(r)(D T g)(r)dr. 

Proof. Since supp/ c V m , one can replace integrals over V m by those over 
the whole space S m ~ R m ( m + 1 )/ 2 . The function g can be multiplied by a smooth 
cut-off function (in necessary) which = 1 on the support of /. Then the ordinary 
integration by parts yields (1.23). □ 

Lemma 1.6. 
(i) For s G V m and z G 5~ , 

(1.24) D +)fl [exp(-tr(s«))] = (-l) m det(z) exp(-tr(sz)). 
(ii) For s G V m and a G C, 

(1.25) D+(| s r-7r m (a)) = |sr- 1 - d /r m (a-l), d = (m + 1)/2. 

Proof. These statements are well known; see, e.g., [Ga, p. 813], [Herz, p. 
481], [FK, p. 125]. We recall the proof for convenience of the reader. The equality 
(1.24) is verified by direct calculation. Furthermore, by (1.20), 

(1.26) T m (P)\s\- fJ = f cxp(-tr(rs))\r\ f3 - d dr 7 Re/3>d-l. 

"P,„ 
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Applying D+ to (1.26) and using (1.24), we get 

D + {\s\-^Y m {ft)) = (-l) m | s r' 3 - 1 r m ( / 3 + 1), Reft>d-1. 
Now we set ft = d — a. Since 

Y m {d - a + l)/T m (d -a) = (-l) m T m (a)/T m (a - 1), 
(1.25) follows for Re a < 1. By analyticity, it holds generally. □ 

Remark 1.7. By changing notation and using (1.9), one can write (1-25) as 

(1.27) D+\a\ a = b(a)\a\ a - 1 , 
where 

(1.28) 6(a) = a(a + l/2)---(a + d-l), d=(m + l)/2, 

is the so-called Bernstein polynomial of the determinant [FK] . It is worth noting 
that b{a) can be written in different forms, namely, 

(1.29) b(a) = (-l) m 6(l - d - a) = 2~"T(2a + m)/T(2a) = 2~ m (2a, to) 
or 

(1.30) b(a) = (-l)"T m (l - a)/r m (-a) = T m (a + d)/T m (a + d - 1). 
Lemma 1.8. For f e T>(V m ) and z G S^, 

(1.31) (LD + f)(z) = dct(z)(Lf)(z). 
Proof. By (1.23), (1.22) and (1.24), 

(LD+f)(z) = J D_, r [exp(-tr(r«))]/(r)dr = dct(z)(L/)(z), 
v„, 
as desired. □ 

1.5. Bessel functions of matrix argument 

We recall some facts from [Herz] and [FK]. The J -Bessel function J v {r), 
r G V mi can be defined in terms of the Laplace transform by the property 

(1.32) f exp(-tr(zr)) J v {r)\r\ v - d dr = Y m {u) cxp(-tr(z- 1 ))dct(z)- ,y , 
v,» 

d=(m + l)/2, z e T m = V m + iS m . 
This gives 

(1.33) — !-- Mr) = — i-^ f cxp(tr(z - rz- 1 ))det(z)-^z, a eV m , 
T rn (v) (2m) N J 

Re z— (To 

N = m(m + l)/2. This integral is absolutely convergent for Rev > to and ex- 
tends analytically to all v £ C. Moreover, the formula (1.33) allows us to extend 
J v {r) in the r-variable as an entire function J u (z), z — (zij) mxm G C m , so that 
X(z)\ z=r = Mr) and 

(1.34) Jvinz^) = J v {z) for all 7 G O(m). 
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Functions satisfying (1.34) are called symmetric. If Ai, . . . , A TO are eigenvalues of z, 
and 

ci = Ai 4- . . . 4- A TO , (72 = A1A2 + . . . + A m _iA m , . . . , a m = Ai . . . A m 

are the corresponding elementary symmetric functions (which are polynomials of 
Zi_j), then each analytic symmetric function of z G C m is an analytic function of 
to variables <j\, . . . , a m . Since for any r G V m and s G V mi the matrices 

s^rs 1 ' 2 , r^sr 1 / 2 

have the same eigenvalues, then 

Mrs) = X(sr) = Ms 1/2 rs 1/2 ) = J v (r 1/2 sr 1 ' 2 ). 

In the following, we keep the usual notation J v iz) for the extended function J v {z). 
For to = 1, the classical Bessel function J v (r) expresses through J v (r) by the 
formula 

1 (T\ v ( r 2 

Mr) = WTT){2) X+1 U 

There is an intimate connection between the Fourier transform and J'-Bessel 
functions. 

Theorem 1.9. [Herz, p. 492], [FK, p. 355] If f(x) is an integrable function 
of the form f(x) = fo(x'x) where /o is a function on V m , then 

(1.35) {^f){y)= J e^{tv{iy'x))h{x'x)dx = fr^j^ fo (*f 

Wl n , m 
where 

(1.36) /o(s) - J J n/2 (rs)\r\ n / 2 - d f Q (r)dr, d=(m+ l)/2 

Vm 

(the Hankel transform of fo). 

This statement represents a matrix generalization of the classical result of 
Bochner, see, e.g., [SW, Chapter IV, Theorem 3.3] for the case to = 1. Since the 
product rs in (1.36) may not be a positive definite matrix, the expression J n i<2,(rs) 
is understood in the sense of analytic continuation explained above. 

Note that one can meet a different notation for the JT-Bcsscl function of matrix 
argument in the literature. We follow the notation from [FK]. It relates to the 
notation Ag(r) in [Herz] by the formula J v (r) — T m (u)Ag(r), S — v — d. More 
information about ^7-Bessel functions and their generalizations can be found in 
[Dib], [FK], [FT], [GK1], [GK2]. 

1.6. Stiefel manifolds 

For n > to, let V n>m = {v G ^Ol n ,m '■ v'v = I m } be the Stiefel manifold of 
orthonormal m-frames in M. n . If n = to, then V n>n — 0(n) is the orthogonal group 
inR n . It is known that dimV^m = m(2n — m — l)/2 [Mu, p. 67]. The group 0(n) 
acts on Vn, m transitively by the rule g : v — > gv, g G 0(n), in the sense of matrix 
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multiplication. The same is true for the special orthogonal group SO(n) provided 
n > m. We fix the corresponding invariant measure dv on V n , m normalized by 

/am_nm/2 
dv= r i /n\ ' 
r m (n/2) 

V n , m 
[Mu, p. 70], [J, p. 57], [FK, p. 351]. This measure is also 0(m) right-invariant. 

Lemma 1.10. [FK, p. 354], [Herz, p. 495]. The Fourier transform of the 
invariant measure dv on V n ^ m represents the J -Bessel function: 



(1.38) 



exp(tr(iy't>))dv = a„ im J n / 2 



v n> 



yy 



Lemma 1.11. (polar decomposition). Let x G 9Jt n . m , n > m. //rank(x) 



m, 



ther 



x = vr ' , 



v e V n>r 



T X X fc I 77i , 



id dx = 2- rn \r\ ( - n - m -V/ 2 drdv. 



This statement can be found in many sources, sec, e.g., [Herz, p. 482], [GK1, 
p. 93], [Mu, pp. 66, 591], [FT, p. 130]. A modification of Lemma 1.11 in terms of 
upper triangular matrices t £ T m (see (1.2)) reads as follows. 

Lemma 1.12. Let x G 9Jt„. m , n> m. J/rank(a;) = m, then 
x = vt, v G V n>m , t G T m , 
so that 

m 

dx = II t" j J dtjj dt* dv, dt* = I I dtij. 

i=i i<3 

This statement is also well known and has different proofs. For instance, it can 
be easily derived from Lemma 1.11 and (1.3); see Lemma 2.7 in [Ru7]. 

Lemma 1.13. (bi-Stiefel decomposition). Let k, m, and n be positive integers 
satisfying 

1 < k < n — 1, 1 < m < n — 1, k + m < n. 

(i) Almost all matrices v G V n . m can be represented in the form 



(1.39) 


v = 


a 

u{L m -a'af/ 2 


i 


a G 9^fc,m, U G V n -k,m, 


so that 






(1.40) 


l« 


v)dv = / d 


H(a) 


/'( 


a 

_ w^-a'a) 1 / 2 _ 



du. 



V„, m 0<a'a</ 7Tl V„_ fci7Tl 

d/x(a) = |/ m - a'a\ s da, 6 = (n - fc)/2 - d, d = (m + l)/2. 
(ii) //, moreover, k > m, then 



(1.41) 



/ 



wr l/2 



f(v)dv = / dis(r) / dui 
v„, m V fc , m V,. 

&/(r) =2- m |r| 7 |/ m -r| ,5 dr, 7 = jfc/2 - d 



du, 
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Proof. For m=l, this is a well known bisphcrical decomposition [VK, pp. 12, 
22]. For k = m, this statement is due to [Herz, p. 495]. The proof of Herz was 
extended in [GR] to all k + m < n. For convenience of the reader, we reproduce 
this proof in our notations which differ from those in [GR] . 

Let us check (1.39). If v = , € V n ^ n , a e %ftk,m, b £ VJl n -k,m, then 

/,„ = v'v = a'a + b'b, i.e., b'b = I m — a'a. By Lemma 1.11, for almost all b 
(specifically, for all b of rank m), we have b — u(I m — a'a) 1 / 2 where u £ V n -k,m- 
This gives (1.39). In order to prove (1.40), we show the coincidence of the two 
measures, dv and dv = \I m — a! a\ b dadu. Following [Herz], we consider the Fourier 
transforms 

Fi{y) = / cxp(tv (iy'v))dv and F 2 {y) — / exp(tr(iy'v))dv, 
v n , m v n>m 

y E DJln^m, and show that F\ = F 2 . Let 



y 



2/1 

V2 



i(I m - a 1 a) 1 ' 2 



My) 



where 

2/i E 97lfc, m , y 2 E ®l n _fe, m ; a e Mk, m - 

Then y'v = y[a + y' 2 u(I m — a'a) 1 ' 2 , and we have 

cxp(tr(iy 1 a))|/ m — a'a\ da 

a f a<Im 

cxp(tr(iy' 2 u(I m — a'a) ))du. 

v n _k, m 

By (1.38), the inner integral is evaluated as 

&n-k,m J(n-k)/2(jy2y2{Im ~ a'a)). 

Thus 

F 2 (y) = / cxp(tr(iyia))</?(a'a)(ia, 

a'a<Irn 

tp(r) = a n -k,m\I m - r\ (n - k)/2 - d J (n -k)/2(^y 2 y2(I m - r)). 

The function F 2 (y) can be transformed by the generalized Bochner formula (1.35) 
as follows 

i m 



Fi(y) 



km/2 
n u n —k' 



x J (n -k)/2{- A y'^2{I m - r))\I m - r|("- fc )/ 2 - d dr. 



The last integral can be evaluated by the formula 

s 

(1.42) f J a (pr)\r\ a - d J p (q( S -r))\s-rf- d dr = B m (a,f3)J a+fj ((p+q) S )\ S \ a +f i - d , 
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Rea,Ref3>d—l, d= (m + l)/2, p, g g "P m , seV m , 

cf. [Herz, formula (2.6)]. Applying this formula with a = fc/2, (3 — (n — k)/2, and 
s = I m , we obtain 

7T km/2 (J f k n — k\ 1 

F2iy) = r m (fc/2)'" l Bm {r^J 1^/2(4(^1 + ^2)) 

= Vn,mJn/2(jy'y)- 

By (1.38), this coincides with F\{y), and (1.40) follows. The equality (1.41) is a 
consequence of (1-40) by Lemma 1.11. □ 

1.7. Radial functions and the Cayley-Laplace operator 

The notion of a radial function usually alludes to invariance under rotations. 
In the classical analysis on W 1 the radial function f(x) is virtually a function of a 
non-negative number r = \x\. In the matrix case the situation is similar but now r 

is a positive semi-definite matrix. 

Definition 1.14. A function f(x) on SDT„. m is called radial, if it is 0(n) left- 
invariant, i.e., 

(1.43) f{ix) = f{x), V 7 eO(4 

If / is continuous, this equality is understood "for all x". For generic measurable 
functions it is interpreted in the almost everywhere sense. 

Lemma 1.15. Let x G dJl n , m . Each function of the form f(x) = fo(x'x) is 
radial. Conversely, ifn>m, and f(x) is a radial function, then there exists fo(r) 
on V m such that f(x) — fo(x'x) for all (or almost all) x. 

Proof. The first statement follows immediately from Definition 1.14. To prove 
the second one, we note that for n > m, any matrix x G 9Jt„ ;TO admits representation 
x = vr 1 ' 2 , v G V n , m , r = x'x G P m ; see Appendix C, 11. Fix any frame wo G V n ,m 
and choose 7 G 0(ri) so that 7^0 = v. Owing to (1.43), 

f{x) = f{vr 1 ' 2 ) = /( 7 «or 1/2 ) = f^r 1 ' 2 ) ^ / (r), 
as desired. Clearly, the result is independent of the choice of vo G V n ,m- O 

The Cayley-Laplace operator A on the space of matrices x — (xij ) G $H n . m is 
defined by 

(1.44) A = dct(<9'<9). 

Here, d is an n x m matrix whose entries are partial derivatives d/dxij. In terms of 
the Fourier transform, the action of A represents a multiplication by the polynomial 
(-l) m P(y), y G 9Jl„, m , where 



P(y) = \y'y\ = dct 



yi-yi ■ ■ yi-y-n 



Vm-Vl ■ ■ Vm-ym 

j/i, ... , y m are column- vectors of y, and " • " stands for the usual inner product in 
R n . Clearly, P(y) is a homogeneous polynomial of degree 2m of nm variables yij, 
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and A is a homogeneous differential operator of order 2m. For m = 1, it coincides 
with the Laplace operator on M. n . 

Operators (1.44) and their generalizations were studied by S.P. Khekalo [Kh]. 
For m > 1, the operator A is not elliptic because P(y) = for all non-zero matrices 
y of rank < m. Moreover, A is not hyperbolic, [Ho, p. 132], although, for some 
n, m and £, its power A enjoys the strengthened Huygens' principle; sec [Kh] for 
details. 

The following statement gives explicit representation of the radial part of the 
Cayley-Laplace operator. 

Theorem 1.16. [Ru7]. Let f (r) e C 2m {V m ), f(x) = fo(x'x), d = (m+ l)/2. 
Then for each matrix x e 9Jl n ,m of rank m, 

(1.45) (Af)(x) = (Lf )(x'x), 
where 

(1.46) L = A m \r\ d - n/2 D + \r\ n/2 - d+1 D+, 
D + being the operator (1.21). 

Example 1.17. Let f(x) = \x\^, \x\ m = dct^-r'a;) 1 / 2 . By Theorem 1.16, we 
have (A/)(a>) = ip(x'x), where ip(r) = A m \r\ d - n l 2 D + \r\ n l 2 - d+1 D+\r\ x / 2 . This 
expression can be evaluated using (1.27): 

ip(r) = A m h{\/2)\r\ d - n l 2 D + \r\ { - n+x ^ 2 - d 

= 4 m 6(A/2)6((n + A)/2-d)|r| A / 2 - 1 . 
Thus, we have arrived at the following identity of the Bernstein type 

(1.47) &\x\l = B{\)\x\ x m 2 , 

(cf. [FK, p. 125]) where, owing to (1.28), the polynomial B(X) has the form 

m— 1 

(1.48) B{\) = {-l) m Y[ (A + »)(2 -n-X + i). 

i=0 

An obvious consequence of (1-47) in a slightly different notation reads 

(1.49) A k \x\^ +2k - n = B k (a)\x\^ n , 
where 

m — 1 k— 1 

(1.50) B k (a) = ~Yl Y[(a-i + 2j)(a-n + 2 + 2j + i) 

i=0 j=0 

= B k {n-a-2k). 



CHAPTER 2 



The Garding-Gindikin fractional integrals 



2.1. Definitions and comments 

In this section, we begin detailed investigation of fractional integrals associated 
to the cone V m of positive definite to x to matrices. These integrals were introduced 
by L. Garding [Ga] and substantially generalized by S. Gindikin [Gil]. 

Let f(r) be a smooth function on V m which is rapidly decreasing at infinity and 
bounded with all its derivatives when r approaches the boundary dV m - Consider 
the integral 

(2.1) I f {a)= J f{r)\r\ a - d dr, d=(m + l)/2. 

Vm 

If f(r) = exp(— tr(r)) then 

m— 1 

I f (a) = T m (a) = ^(™-D/ 4 11 T(a-j/2) ; 

3=0 

see (1.6), (1-7). In the general case, by passing to upper triangular matrices and 
using (1.3), we have 

2a — 1 it I ,2a — 2 i, I ,2a— m 



If( a ) — / tlA dt\,l I ^2% dt2,2 ■■■ tm,7Ti n f(tl,l,---,t m , m )dt 



= 2- J y?- 1 d Vl J yT ilA dy 2 ...J y^ m+1 ^f(y^, . . . , yV*) dy m , 



f(h,i,...,t m , m ) = 2 m j f(t't)dU, dU=l[dt iij . 

K m(m-l)/2 l <3 

The function / extends to the whole space M™ 1 as an even function in each argument, 
and the function 

f ( yi ,..., ym )^2- m f(y 1 1 /2 ,..., y l/ l 2 ) 
belongs to the Schwartz space §(M m ). Hence If (a) can be regarded as a direct 
product of one-dimensional distributions 

(2.2) l f (a)= I f[(y 3 f + - U+1)/2 Jo(yi,---,y m )) ■ 

It follows [GShl, Chapter 1, Section 3.5] that the integral (2.1) absolutely converges 
if and only if Re a > (to — l)/2 and extends as a meromorphic function of a with 
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the only poles (to — l)/2, (m — 2)/2, .... These poles and their orders coincide 
with those of the gamma function T m (a) so that I{a)/Y m (a) is an entire function 
of a. 

These preliminaries motivate the following definitions. We first consider "good" 
functions / E T>(V m ) which are infinitely differentiable and compactly supported 
away form the boundary dV m . Then we proceed to the case of arbitrary locally 
integrablc functions. 

Definition 2.1. Let / e T>{P m ), a E C, d = (m + l)/2. The left-sided 
Gdrding- Gindikin fractional integral is defined by 



(2.3) 



m)(s) 



i 



f(r)\s 



\a— d 



r m (a) 

o 
(I« +i Dif)(s) 



dr 



if Re a > d — 1, 



if d - 1 - 1 < Rea < d - £; 

£=1,2,.... 



Here, s E ? m , D + is the differential operator (1.21), and we integrate over the 
"interval" (0, s) — {r : re V m , s - r E V m }. 

Definition 2.2. For / E T>(P m ) and aeC, the right-sided Gdrding- Gindikin 
fractional integral is defined by 



(2.4) 



1 



(I-f)i-s) = 



r TO (a) 

s 

{I1 +i Dif)(s 



f(r)\r 



x-d 



dr 



if Re a > d — 1, 



if d-l-e < Rea<d-£; 
£ = 1,2,.... 



Here, s is a positive semi-definite matrix in V m , D- is defined by (1.22), and J 
denotes integration over the shifted cone s + V m = {r : r e V m '■ T — s e ? m }. The 
vertex s of this cone may be an inner point of V m or its boundary point. 

In the rank-one case m = 1, when ? m is a positive half-line, (2.3) and (2.4) are 
ordinary Ricmann-Liouvillc (or Abel) fractional integrals [Rul], [SKM]. 

Remark 2.3. Analogous definitions can be given for functions on the ambient 
space S m D Vm, for instance, for / E S(<S m ). We do not do this for two reasons. 
Firstly, because our purposes are exclusively connected with analysis on the cone 
Vm, and the space T)(V m ) fits our needs completely. Secondly, dealing with the 
space T>(V m ), we can still reveal basic features of our objects and give much simpler 
proofs then for other classes of functions. 

According to Definitions 2.1 and 2.2, it is convenient to regard the complex 
plane as a union C = U^ f^, where 

( {a: Rea> d-1} if i = 0, 

(2.5) n e = <^ 

[ {a : d-l-e < Re a < d - £} if £ = 1,2,.... 
Lemma 2.4. For f E r D('P m ) ) the integrals (I±f)(s) are entire functions of a. 
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Proof. Let us consider the integrals (7"/)(s). Given an integer j > 0, we set 
Fj( a ) = 1+ " ' D+f- This function is analytic in the half-plane Rea > d — 1 — j. 
Owing to uniqueness property of analytic functions, it suffices to show that Fj (a) — 
I+f on each strip Qg provided that j = j(£) is large enough. For a G 0^, we have 
a + j — d > j — £ — 1, and therefore, by (1.23) and (1.25), 

Fj{a) = ^^J( S -r T+ +^ + mr)dr 
s m 



1 



s 



&- e r {s-r)l +i -\D{f){r)dr 
\s-r\ a+i - d {DU)(r)dr 



o 

= (I« +e Dif)( S ). 

By Definition 2.1, this means that Fj(a) — I"f for a G fie, and we are done. For 
operators /" the proof is similar. D 

Lemma 2.5. Let f € T>(V m )- Then 

(2.6) D> ± I£f = I£D>±f = l£- j f, 

for any aeC and j G N. 

PROOF. Fix j G N. For Rea > j + d - 1, the equality I±D 3 ± f = I±~ j f can 
be obtained using integration by parts (see the proof of Lemma 2.4). For all a G C 
it then follows by analytic continuation. Let us show that D 3 ± I±f = I^T J f . Let 
Rea- j G Qt ; see (2.5). If £ = 0, then 

S 



= f^/ D +>-)r d /(^ 

= {Di + I«f){s). 

If £ > 1, then, by Definition 2.1, J? -J '/ = I? - - 7 £*+/• By the previous case, this 
coincides with _D^_/" + !?+/ = D J + I"f (here we used Definition 2.1 again). For the 
right-sided integrals the argument follows the same lines. □ 

Some historical notes and comments are in order. Fractional integrals similar 
to (2.3)-(2.4) and associated to the light cone were introduced by M. Riesz [Ril], 
[Ri2] in 1936. The investigation was continued by N.E. Frembcrg [Frl], [Fr2], and 
by Riesz himself in [Ri3], [Ri4]; see also B.B. Baker and E.T. Copson [BC], and 
E.T. Copson [Co]. A key motivation of this research was to find a simple form 
of the solution to the Cauchy problem for the wave equation. M. Riesz' argument 
was essentially simplified by J.J. Duistermaat [Dui]; see also J. A.C. Kolk and V.S. 
Varadarajan [KV]. 

The results of Riesz were extended by L. Garding [Ga] (1947), who replaced 
the light cone (of rank 2) by the higher rank cone V m of positive definite symmetric 
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matrices, and studied the corresponding hyperbolic equations. 1 The Riesz-Garding 
fractional integrals arc very close to those in (2.3) but do not coincide with them 
completely. The difference is as follows. In the above-mentioned papers, mainly 
devoted to PDE, the upper limit s in (2.3) varies in the whole space R^, N = 
m(m + l)/2 (not only in V m ), and the domain of integration is bounded by the 
retrograde cone s — V m and a smooth surface on which the Cauchy data are given. 
For s G V m , this means that the part of the domain of integration in (2.3) containing 
the origin is cut off smoothly. 

Further progress is connected with fundamental results of S.G. Gindikin [Gil] 
(1964), who developed a general theory of Ricmann-Liouvillc integrals associated 
to homogeneous cones. The light cone of Riesz and the cone V m of Garding fall 
into the scope of his theory. In the case of V m , Gindikin's fractional integrals are 
convolutions of the form 



(a) 



(2.7) J«/=_±_*/, feS(R N ), N 



where the function s" d equals \s\ a d for s £ V m , zero otherwise, and s°_ d equals 
(— s)°^T d . Operators (2.7) are defined on the whole space R*, and can be treated 
using the Fourier transform technique and the theory of distributions; see [Gil], 
[VG] , [Rab] , [Wa] , [FK] . See also [Rich] concerning application of such operators 
in multivariate statistics. 

Definitions 2.1 and 2.2 are motivated by consideration of Radon transforms of 
functions of matrix argument in the next chapters. It is important that the variable 
of integration r and the exterior variable s do not range in the whole space and 
are restricted to V m . In the following, the class of functions / will be essentially 
enlarged. We shall define and investigate fractional integrals I±f of arbitrary locally 
intcgrablc functions / with possibly minimal decay at infinity. The main emphasis 
is made upon a belonging to the Wallach-likc set W m = {0, |, 1, |, . . . , ^j^-} U 
{a: Rea> 2f±}, cf. [FK, p. 137]. 

Thus, one can see that most of the facts for fractional integrals from [Ga] and 
[Gil] cannot be automatically transferred to our case, and a careful consideration 
is required. The case a = fc/2, fc e N, is of primary importance because it arises in 
our treatment of the Radon transform. 

2.2. Fractional integrals of half-integral order 

An important feature of the higher rank fractional integrals I±f is that for 
a = fc/2, fc = 0, 1, . . . , m — 1, they are convolutions with the corresponding positive 
measures supported on the boundary dV m - In the rank-one case m = 1, this 
phenomenon becomes trivial (we have only one value a — corresponding to the 
delta function). For m > 1, when dimension of the boundary &P m is positive we deal 
with many "delta functions" supported by manifolds of different dimensions. This 
phenomenon was drawn considerable attention in analysis on symmetric domains; 
see [FK, Chapter VII], [Gi2], [Ishi], and references therein. 

Our nearest goal is to find precise form of J ± / for fc = l,...,m— 1. We start 
with the following simple observation. For k > to, by Lemma 1.11 and (1.37), one 



Prof. Lars Garding kindly informed the second-named author that his construction of 
fractional integrals was inspired by close developments in statistics. 
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can write 








(4 /2 /)(*) = 


s 

/ f(" r\\r\ k / 2 ~ d r!r 




- T m (k/2)J nS r)|r| dr 




(2.8) 




= TT- km / 2 f f{s-oj'oj)duj, 




(i- /2 fM = 


/ f( n 1 r 'i| r | A; / 2 - rf / 7 r 




- T m (k/2)J K s + r ^ dr 


(2.9) 




= TT- km ' 2 [ f(s + u'u)du. 



3Jt fc , m 

The expressions (2.8) and (2.9) are meaningful for all k G N, and it is natural 
to conjecture that for k — 1, . . . , to — 1, the integrals J ± ' / are represented by (2.8) 
and (2.9) too. Let us prove this conjecture. We start with the left-sided integrals 
I, f and make use of the Laplace transform. 

Lemma 2.6. For f e T>(T m ) and a e C, 

(2.10) (LI*f)(z)=dct(z)- a (Lf)(z), Rez>0. 

PROOF. Let a e Vti ; see (2.5). For I = 0, (2.10) is an immediate consequence 
of (1.19) and (1.20). The desired result for t > 1 follows by Definition 2.1 and 
(1.31): 

(LIlf)(z) = (LI« +i D e + f)(z) = det(z)- a -\LD{f)(z) = dct(z)- Q (L/)(z). 

D 

Corollary 2.7. For / e D(T> m ), 

(2.11) (Dif)(s) = (I+ e f)(s), £ = 0,1,2,.... 

PROOF. By (1.31) and (2.10), {LD e + f)(z) = (LI+'f)(z), Re z > 0. Since, 
by Definition 2.1, I+ e f = lfD e + d f if d = (to + l)/2 is an integer, and /+*/ = 

J_i_ D + / otherwise, the result follows owing to the uniqueness property of 

the Laplace transform (see Lemma 1.3). 

□ 

Theorem 2.8. Iff e 1>(V m ), then for all keN, 

(2.12) (I k + /2 f)(s) = 7r- km / 2 J f(a-u/u)du. 

Moreover, 

(2-13) (£/)(*) = /(*). 
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PROOF. Denote by tp(s) the right-hand side of (2.12). By changing the order 
of integration, we have 

{L<p){z) = n~ km ' 2 J exp(-tr(z S ))d S J .f{s-y'y)dy 

Vm {yeStfc,™: v'v<s} 

(2.14) = n- km / 2 (Lf)(z) J exp(-ti(zy'y))dy. 

M k , m 

The integral in (2.14) is easily evaluated: if z = r G V m , we replace y by yr -1 / 2 
and get 

/ exp(— tr(ry'y))dy = \r\~ ' / exp(— %v{y'y))dy 

m h , m mt k . m 

- M- fe / 2 



/ exp(-y 2 ;1 - ... - s/fe )m )dy 



7r fem/2i|-fe/2_ 



By analytic continuation, 

exp{-\x(zy'y))dy = 7r fem/2 det(,z)- fc/2 , i?e z > 0. 



%.„ 



Thus, (L<p)(z) = det(z)- fc / 2 ( J L/)(z), and (2.10) yields {Ltp)(z) = {LI^ 2 f){z), 
Rez > 0. Since for each cto G "P m , the integrals 

exp(-ti(aos)\<fi(s)\ds, / exp(-tr(<j s)|(/ + / 2 /)(s)|ds 



are finite (both are dominated by |<7o k ^ 2 (L\f\)(ao)), by Lemma 1.3 we get <p(s) 
(I + ' f)(s), as desired. 

Remark 2.9. For < k < m, the integral (2.12) can be written as 
(2.15) (I k + /2 f)(s) = c k , m \ S \ k / 2 J dvJf{8-s 1 ' 2 v<p/8 1 ' 2 )\q\<- m - k - 1 V 2 dq, 

V m , k 



D 



Cfe, 



„—km/2c\ — k 



Indeed, if we transform (2.12) by setting u> = hs 1 / 2 , where h G Tlk, m , h'h < I m , 
and pass to polar coordinates, we obtain (2.15). 

The integral (2.12) can also be written as the Laplace convolution 

s 

(2.16) {lTf){s)= ff(8-r)dn k (r), 



G = GL(m,R), e k 



edP„ 
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where \xk is a positive measure defined by 

(2.17) ( Mfc ,^) = 7r- fcm / 2 I i;(uj'uj)duj, VeC c (5 m ). 

m k , m 

Here, C c {S m ) denotes the space of compactly supported continuous functions on 
S m . For fc > to, Lemma 1.11 yields 

(Mfe ' ^ = f^k/2j J ^ r )l r | fe/2 ^ dr > d=(m + l)/2. 

In order to clarify geometric structure of /Ufc for fc < to, we denote 

(2.18) A fe = {s : s e P^", rank(s) = fc}, fc = 0, 1, . . . ,m- 1, 

Ifc 



The manifold Afe is an orbit of ek under G in S m , i.e., Afe = Gefe. Indeed, 
rank(g'efeg) = k for all g G G, and conversely, each matrix s of rank k is rep- 
resentable as s — g'e^g for some g € G; see, e.g., [Mu, A6 (V) and Theorem 
A9.4(ii)] . The closure Afe is a union of Ai, . . . , Afe. Since rank(o/w) < k < to, then 
(fik,ip) = for all ip supported away from Afe = Gefe, and therefore supp/Xfe = Afe. 
Note also that /zk(Afc_i) = because 

ip(s)d/ J Lk(s) = / i/)(w'w)(kj = 0, 

A fc _i 2t fc _i 

2lfc_i = {o> 6 Sftfc.m : rank(w) < fc — 1} (the set 2lfc_i has measure in 9Jtfc, m ). 
Thus the following statement holds. 

Theorem 2.10. For < fc < m, /,' / is ifte Laplace convolution (2.16) with 
a positive measure /ife defined by (2.17) and such that (a) supp/Xfe = Ge^, and (b) 
,Ufe(Gefe_i) = 0. 

Let us consider the right-sided integrals /"/. They can be expressed through 
the left-sided ones as follows. 

Lemma 2.11. For f e T>{V m ) and a e C, 

(2.19) (IZf)(s) = Isr^g^s- 1 ), g(r) = \r\- a ~ d f (r' 1 ) . 
In particular, 

(2-20) (I a _.f)( S )=f(s). 

Proof. Since the integrals (I+f)(s) and (i"/)(s) are entire functions of a (see 
Lemma 2.4), it suffices to prove (2.19) for Rea > d — 1. This can be easily done 
by changing variables according to Lemma 1.1 (iii). □ 

Corollary 2.12. For f e T>{V m ) and I e N. 

(2.21) (Dif)(s) = H-^OD^X*- 1 ), g(r) = M'""/^- 1 ). 

PROOF. The formula (2.21) follows from (2.11) and (2.19). 

□ 

Now we can justify (2.9) for all fc e N. 



34 2. THE GARDING-GINDIKIN FRACTIONAL INTEGRALS 

Theorem 2.13. Iff e T>(V m ), keN, then 

(2.22) {I- /2 .f)(s) = ir- km ' 2 f f(s + co'uj)du). 

m k , m 

Proof. According to (2.19), 

(2.23) (Il /2 f)(s) = \s\ k /*- d (I k + /2 g)(s-i), g(r) = \r\- k /^ d /(r" 1 ). 
By (2.15), (I, g)(s^ 1 ) is represented as 



c k , m \s\~ k/2 / dv g(s- 1 -s- 1 / 2 vqv's-^ 2 )\q\^ m - k - 1 ^ 2 dq 

V m , k 



Cfe,i 



,i^/4/((,--,-v.-'-)-) ^_^ . 



Using the property 

(2.24) |/ m -wgw'| = |/ fc -g|, ge^, »eV m i, 

(see [Mu, p. 575] or Appendix C, 1 ), this can be written as 

ik 
c k:m \ S \ d J dv j f^'\l m - vqvT 1 ^) ^T^Zf 
v m . k 



//• „|(m— fc— 1)/2 J 

dlj f{ S l/2 l(I m v qv> )-Ws 1/2 ) lQ h _ qlk/2+ f , 
0(m) 



VO 




h 



ev r 



m,k • 



One can readily check that 

(I m -vaqv^y 1 = 

u ■** ~~ SZ J L " 

After changing variable (Ik — q)^ 1 — » 9, the integral (I, g)(s~ 1 ) becomes 



l-m—k U 

h-q 



■*-m—k U 

o (4 - g)" 1 



Cfc, m cr m ,fe|s| d / d-f / / ( s 1/2 7 

O(m) / fc 



*m—k U 

g 



7' S 1/2 )|4-9l (m - fe - 1)/2 ^, 
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and therefore (replace q by Ik + q) , 

(I^gKs- 1 ) = c Km \s\ d J dvf f(8 1 / 2 (I m +v<p/)a 1 ' 2 )\q\t m - k - 1 V 2 dq 

v m , k v k 

= c Km 2 k \s\ d J fis + s^yy's^dy (y = S - 1/2 z) 

m m , k 

= c Km 2 k \s\ d - k/2 J !{s + zz')dz 

m m , k 

Wl k , m 
Owing to (2.23), this gives the desired equality. □ 

Remark 2.14. For < k < m, the integral (2.22) can be written in polar 
coordinates as 

(2.25) (I- /2 .f)(s) = C k , m j dvjf(vqv' + s)\q\^- k -^l 2 dq, 

V m , k v k 

—km/2cy — k 

*-<k,m — I ^ 

Furthermore, 

(I- /2 f)(s)= J f(s + r)dn k (r), 
v m 
\i k being a positive measure defined by (2.17). 

2.3. The Garding-Gindikin distributions 

It is instructive to give an alternative proof of the formula (2.22). This proof 
follows the argument from [FK, p. 134] and is of independent interest. It can be 
useful in different occurrences. Consider the Garding-Gindikin distribution 

(2.26) g a (f) = -±- f f(r)\r\ a - d dr, d=(m + l)/2, 

F m (a) J 
v m 

where / belongs to the Schwartz space §(S m ). The integral (2.26) converges abso- 
lutely for Re a > d — 1 and admits analytic continuation as an entire function of a 
so that 

(2.27) g (f) = /(0), 

see [FK, pp. 132-133]. The following statement implies (2.22) and extends it for 

/ e §{S m ). 

Lemma 2.15. For f e §(S m ) and < k < m, 

(2.28) G k/2 (f) = n- km / 2 j f{u/u)du. 

M k _ m 
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PROOF. Let us transform Ga(f) to upper triangular matrices. By (1.3), 
2 m 



Ga{f) = 



r m (a) 



f(t't) II f M ~* II dU >i Rea>d-1. 

i— 1 i<j 



We write t = a + b, where a = (aij) and b = (bij) are upper triangular matrices 
so that the lower n — k rows of a and the upper k rows of b consist of zeros. We 
denote by A and B the sets of all matrices of the form a and b, respectively. Since 
t't = a' a + b'b, then 



(2.29) g a (f) 

where 

g a (aa) = 



2 fc r m _ fc (a-fc/2) 
r TO (a) 



/ 5a(a'a)J| a M * II da ^ ' 



jm—k 



r m _fe(a - fc/2) 



m — k 



f{a' a+ b'b) n fe+f^n*^.^- • 



»=1 



»<J 



Note that g a (a'a) = G a -k/2 I / 






is the Garding-Gindikin distribution 



acting on the (•) matrix variable belonging to S m ~k- By (2.29), a (f) is a direct 
product of two distributions which are analytic in a. By taking into account (2.27), 
we get <7fc/2(o'a) = /(o'a), and therefore, 



(2.30) 

Here, by (1.7), 



<?fc/2(/) = c f f(a'a)f[a^l[da h3 . 

J A »=1 »<j 

2 fe r m _ fe (a - fc/2) 2 fc 7r fc ( fc - m )/ 2 



c = 2 K lim 

a^fe/2 



T m (a) 



r fc (A;/2) 



This representation was established in [FK, p. 134]. Let us show that it coincides 
with (2.28). We replace uj in (2.28) by [r?,C], where 77 G 9Jt fejfc , C e 9Kfe,m-fc- Then 



7T-W2 J /( W ' W )du; = 7r- fcm / 2 / *7 / /([^ f^ 

wi k , m m k , k m k , m - k 

( set r\ = jq, 7 G O(k), q £ T k and use Lemma 1.12) 



d( 



n- km/2 a k , k jl[q^l[ dhj 

rp i—1 i<j 



f 



m k 



qq qy 
y'q y'y 



(1.37) 2 fc 7r fe (' c " m )/ 2 
T k (k/2) 



i— 1 i<_7 



where a 



<? 2/ 




This proves the statement. 



rfy 



□ 
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:S7 



2.4. Fractional integrals of locally integrable functions 

In the previous sections, we studied fractional integrals I±f assuming / is very 
nice, namely, / e T>(V m )- Below we explore these integrals for arbitrary locally 
integrable functions / provided that a belongs to the Wallach-like set 



(2.81) W„, = «{0,^1,|,. 



m — 1 I f to — 1 
U < a : ite a > 



The key question is for which / the integrals I±f do exist. We start with the 
left-sided integrals. 

Definition 2.16. For a locally integrable function / on V m and a e W TO , we 
define 



8 

-!— / f{r)\s-r\ a - d dr HRea>d-l, 
n(a) J 



(2.32) (!«/)(*) = 



7r- fem /2 / /( s - w ' w )du; ifa = fc/2. 

{uJ^yj[ fcjTT1 : uj / uj<s} 



Here, s G T^m, d = (m + l)/2, and fc — 1,2, . ..,m — 1. For a = 0, we set 
(7°/)(s) = /(s)- 

This definition agrees with consideration in Sections 2.1 and 2.2. We recall that 
the second line in (2.32) coincides with the first one if k > to, a = k/2. 

Theorem 2.17. If f £ L\ oc (V m ) and a e W TO) then (I"f)(s) converges abso- 
lutely for almost all s G V m . 

PROOF. It suffices to show that the integral L (I"f)(s)ds is finite for any 
a e V m provided that a E W TO and / e L] oc {V m ) is nonnegative. For a > d — 1, 
changing the order of integration, and evaluating the inner integral according to 
(A.l), we have 



(I«f)(s)d S = 



1 



r m (a) 



o 

r m (d) 



T m (a + d) 



f{r)dr / \s-r\ a - d ds 



f(r)\a-r\ a dr 



T m (d)\a\ a , , 

l m (a + oj 
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(note that \a — r\ < \a\; see, e.g., [Mu, p. 586] ). For the second line of (2.32), by 
changing the order of integration, we obtain 



{ll /2 f){s)ds = TT~ km / 2 I dcu I f(s-uj'uj)ds 



W'w<fl LO'UJ 

a—uj uj 

,v '"/ 2 f dcu 



f(r)dr 



where 



as required. 



uj'uj<a 



a 

< *- km/2 c a [ f(r)dr, 



(A.7) ir km / 2 T m {d) k/2 
r m (k/2 + d) 



a 



The following equality can be easily obtained by changing the order of integra- 
tion and using (A.4)-(A.6): 



(2.33) 



(/?/)(«) ds r ro ( 7 -a) f f(r)dr 



|e/ m + s|T T m ( 7 ) J \el m + r\-y- a 



aeW m , Re (7 - a) > (to - l)/2, e = 0, 1. 
Let us consider the right-sided integrals /"/. An idea of the following definition 
is the same as above. 

Definition 2.18. For a locally integrable function / on V m and a £ W TO , we 
define fractional integrals (i"/)(s), s £ P m , by 



(2.34) (!«/)(*) = 



— l — I f{r)\r-s\ a - d dr HRea>d-l, 
T m (a) J 

S 

TT- km / 2 / f(s + uj'u)duj if a = k/2. 



For a — 0, we set (J° /)(s) = /(«)■ As before, here we assume d = (m + l)/2 and 
fc = 1, 2, . . . , m — 1. 

Note that unlike Definition 2.16, here s may be a boundary point of V m - The 
following statement extends Lemma 2.11 (on interrelation between /" and /") to 
locally integrable functions. 

Theorem 2.19. Let a £ W m; d = (to + l)/2. Suppose that f is a measurable 
function on V m satisfying 



(2.35) 



f \r\ Rea - d \f(r)\dr < oo, Vi? £ 7> r , 
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or, equivalently, 

(2-36) g(r) ee \r\-<*~ d f^ 1 ) e L} oc {T m ). 

Then the integral (I"/)(s) defined by (2.34) converges absolutely for almost all 
s E V m and 

(2.37) (I»f)(s) = Isr^gXs- 1 ). 

Proof. Equivalence of (2.35) and (2.36) is obvious in view of Lemma 1.1 
(iii). In fact, we have already proved (2.37) provided that either side of it exists 
in the Lebesgue sense. This was done in Lemma 2.11 for Re a > d — 1, and 
in Theorem 2.13 for a — k/2, < k < m. By Theorem 2.17, the condition 
g{r) = |r| _a /(r _1 ) G i; 1 oc ("Pm) guarantees finiteness of the right-hand side of 

(2.37) for almost all s. This completes the proof. 

□ 

Remark 2.20. Condition (2.35) is best possible in the sense that there is a 
function / > for which (2.35) fails and (/"/)(s) = oo. Let 

(2.38) fx(r) = \I m +r\- x/ \ 
where X < 2a + m — 1, a is real. Then for a > d — 1, 

oo R- 1 

\r\ a - d fx(r)dr= f \r\ x ' 2 - a - d \I m + r\- x ' 2 dr = oo, 

R 

and by (A. 5), 

(2.39) (r/A)00 = |Jm ^ 5 '" 2 \r\ X/2 - a - d \I m -r\ a - d dr 

r m (a) J 
o 

= 14, + s\ a - x ' 2 B m (a : A/2 - a)/r ro (a) = oo. 
In the case a = k/2, k < m, we have 

(I h : /2 fx)(s) = K- km/2 J h(s + Lu'Lu)dcu 
Wl k , m 

(2.40) = K- km / 2 [ \b + yy'\- x ' 2 dy 1 b = I m + s. 

Wl m , k 
Owing to (A. 6), the last integral coincides with (2.39) (with a — k/2) and is infinite. 
The following formula is a consequence of (2.33) and (2.37): 

(2.41) J (I a f)(s) J 517 '""' ds = rm(7 ~" ) / f(r) , H7 ~' dr , 

1 ' J y - J,K ' |4 + es\t P m ( 7 ) J J K) |4 + er\i-° ' 

aeW m , d={m+l)/2, Re (7 - a) > (to - l)/2, £ = 0,1. 
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2.5. The semigroup property 

By the Fubini theorem, the equality (A.l) yields 

(2.42) iplf = J£ +/3 /, Rea>d-1, Re/3>d-l, 

provided the corresponding integral /? / or /" + ^ f is absolutely convergent. The 
equality (2.42) is usually referred to as the semigroup property of fractional inte- 
grals. It extends to all complex a and /3 if / is good enough and fractional integrals 
are interpreted as multiplier operators in the framework of the relevant Fourier- 
Laplace analysis. This approach allows us to formulate the semigroup property 
in the language of the theory of distributions; see [Gil], [FK], [VG], [Rab] for 
details. 

Below we extend (2.42) to the important special case when a and (3 lie in the 
Wallach-like set W m defined by (2.31). Definitions (2.32) and (2.34) enable us to 
do this under minimal assumptions for / so that all integrals are understood in the 
classical Lebesgue sense. 

Lemma 2.21. Iff e L] oc {V m ), then 

(2.43) iplf = l£ +0 f, a,/3eW m , 

provided the corresponding integral on the right-hand side is absolutely convergent. 
PROOF. In view of (2.42), it suffices to prove that 

Re a > d — 1, 



(2.44) 


jajk/2j = 


r k/2 Ta f r a+k/2 f 
- 1 ± 1 ±J — 1 ± /> 


and 






(2.45) 




jk/2jl/2 , _ j{k+l)/2 , 



where k and £ are positive integers less than m. By changing the order of integra- 
tion, we have 

oo 
_— km/2 r r 

{I1I k J 2 f){s) = -—- \r - s\ a - d dr / /(r + c/w)dw 
F m (a) J J 

Tl k , m 

oo 

-km/2 r r 

= y^Jo) J duJ J /(*)!*-*" ww l dt 



oo 
-km/2 



n 



f(t)h(t - s) dt, 



where 



h(z)= / \z~Juj\ a - d duj. 

The same expression comes out if we transform I_ I"f. By (A.6), 

fcm/2-p I \ 

r m (a + fc/2) 
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and we are done. For the left-sided integrals, the argument is similar. Namely, 



——km 1 2 r r 

(ip k + /2 .f)(s) = j-^ J\s-r\ a - d dr J /(r-cAOdw 

-km/2 f 

r m (a) J 



= {Il +k/2 f){s)- 
Let us prove (2.45). By (2.34), 

(/! / 2/ ,/ 2/)(s) = 7r - m(fe+ , )/2 r duj r f{3/y+ j u + 8)dy . 



The change of variables z 



y 



m, m m e 



G 9yik+£,m yields z'z — y'y + a/a;, and therefore 



(I k _ /2 I i J 2 f)(s) = n -™V°+W / f( z 'z + s)dz = (I^ +i)/2 f)(s). 

Wl k+i , m 
For the left-sided integrals, owing to (2.32), we have 

(l{ k+e)/2 f)(s) = 7r-™( fe +^/ 2 J f(s - z'z) dz. 

{zeVJlk+e,m '■ z'z<s} 

By the Fubini theorem, this reads 

n - m (k+e)/2 f du I f{a-y'y-oj'w)dy 

and coincides with (I + I J f)(s). D 

2.6. Inversion formulas 

Below we obtain inversion formulas for the integrals (I±f)(r), r G V m - For 
our purposes, it suffices to restrict to the case a G W m . We recall that k G N and 
d = (m + l)/2. For "good" functions /, the following statement holds. 

Lemma 2.22. If ip = I%f, f G T>{V m ), a G W m , then 

(2.47) / = Dilt a V>, 

provided that j > a if a = k/2 and j > Re a + d — 1 otherwise. 

Proof. By the semigroup property (2.43), we have I 3 ± a <p = I±f '. Then ap- 
plication of (2.6) and (2.13) (or (2.20)) gives (2.47). □ 

Unfortunately, we are not able to obtain pointwise inversion formulas for I±f 
if / is an arbitrary locally intcgrable function. To get around this difficulty, we 
utilize the theory of distributions. 
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Lemma 2.23. If<p = /£/, f e L] oc {V m ), a e W m , then 

(2.48) / = D> + I> + - a <p, 

for any integer j subject to the condition that j > a if a — fc/2 and j > Rea + d— 1 
otherwise. The differentiation in (2.4-8) is understood in the sense of distributions, 
so that 

(2.49) (/, <t>) = (/£->, dL4>), e D(P ro ). 

In particular, for a = j , 

(2.50) (/, 0) = fo DL0). 
Proof. As above, we have I^ a if = I+f, and therefore, 

(/, ^) ( => (/, j£i>i0) ( => (4/, dl<i>) ( => (/r>, i>i0). 

The equality (1) holds thanks to (2.6), in (2) we apply the Fubini theorem; (3) 
holds owing to the semigroup property (2.44). □ 

Let us invert the equation I"f = <p. Formally, / = D"tp where D" = lZ a , or 
(/, <f>) = (ip, IZ a (j)), (j) € ©("Pm) in the sense of distributions. The next statement 
justifies this equality. 

Lemma 2.24. Let a E W m; and let f be a locally integrable function on V m 
subject to the decay condition (2.35), so that (i"/)(s) exists for almost all s £ V m . 
If if = I"f then the integral 



(<P, I+ a <t>) = J V{r){l+ a 4>){r)dr, </> <E 1){P m ), 

v m 
is finite, and f can be recovered by the formula 

(2.51) / = DZv, 
where D"(p is defined as a distribution 

(2.52) (£>«¥>, 0) = {<P, I+ a 4>)- 
In particular, for a = j , 

(2.53) (/, 4>) = (ip, D%4>). 

Proof. Let 0i(s) = \s\ a - d cj)(s- 1 ). This function belongs to T)(V m )- By (2.19), 

(I+<V)to = \8\- a - d {IZ a <h){s- X )- 

Utp = rif then 

(V, IZ a <t>) = ((/?/)(*), \8\- a - d (II a <f> 1 )(8- 1 )) 

(use Theorem 2.19 with g(r) = |r|- a_d /0 ,_1 )) 
= ((Ilg^s- 1 ), Isr^lZ^iKs- 1 )) 
= ((I2g)(s), (/I Q 0i)(3)) 

( => (g, IZIZ a <t>i) 

= (9, 01 ) 

= (/». 
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Let us comment on these calculations. The equality (1) holds by the Fubini 
theorem. Here we take into account that IZ a (f>i G T)(V m ), because by the second 
equality in (2.6), 

(2.54) IZ a (f>i = I 3 S a D 3 _(t> 1 

if j is big enough. A rigorous justification of the obvious equality I"IZ a (f>i = <fii 
used in (2) is as follows: 

wz-h (2 ^ 4) iziL-'DLh (2 i 3) iLDLh (2 ^ Hh (2 ^ 0) <t>,. 

For a = j, the result follows by Corollary 2.7, according to which I7 J <f> = D J + (f>. 
The proof is complete. □ 



CHAPTER 3 



Riesz potentials on matrix spaces 



Riesz potentials of functions of matrix argument and their generalizations arise 
in different contexts in harmonic analysis, integral geometry, and PDE; sec [Far], 
[Ge], [Kh], [St2], [Shl]-[Sh3]. They are intimately connected with representations 
of Jordan algebras and zeta functions (or zcta distributions) extensively studied in 
the last three decades; see [FK], [CI], [Ig], [SS], [Sh] and references therein. In 
this chapter, we explore basic properties of matrix Riesz potentials which will be 
used in Chapters 5 and 6 in our study of Radon transforms. Numerous results, 
bibliography, and historical notes related to ordinary Riesz potentials on R™ can 
be found in [Rul], [Sa], [SKM]. 

3.1. Zeta integrals 

3.1.1. Definition and examples. Let f(x) be a Schwartz function on the 
matrix space 9Jt„, m , n > m. We denote 

(3.1) |x| m = det(x'x) 1/2 . 

For m = 1, this is the usual euclidean norm on M™. If to > 1, then |x| m is the 
volume of the parallelepiped spanned by the column- vectors of the matrix x [G, p. 
251]. Let us consider the following distribution 



(3.2) Z(f,a-n)= / f(x)\x\^ n dx, a e C. 

SDt„, m 

For Re a > to — 1, the integral (3.2) absolutely converges, and for Re a < m — 1, it 
is understood in the sense of analytic continuation (see Lemma 3.2). Integrals like 

(3.2) are also known as the zeta integrals or zeta distributions. 

The case n = m deserves special consideration. In this case, we denote $H m = 
yftm.m and define two zeta integrals 

(3.3) Z+(f,a-m) = f f(x) \dct{x)\ a - m dx (=Z(f,a-m)), 

(3.4) Z-{f,a-m) = f f(x) |dct(x)| a - m sgndet(x) dx. 

We call (3.4) the conjugate zeta integral (or distribution) by analogy with the case 
a = 0, m = 1, where 



2L(/,-l)=p.v. J 



CO 

m dx . 
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A convolution of / with the distribution p.v.- represents the Hilbert transform 
[Ne], [SW], and is called a conjugate function. 

Example 3.1. It is instructive to evaluate zeta integrals for the Gaussian func- 
tions. Let 

(3.5) e(x) — exp(— tr(a/a;)). 
By Lemma 1.11, for Re a > m — 1 we have 

(3.6) Z(e, a-n) = 2~ m a n , m J \r\ a ' 2 - d exp(-tr(r))dr = c„, m r m (a/2), 
where <J n ,m is the constant (1.37), d = (to + l)/2, 

f nm/2 



n > m. 



""" r m (n/2) 

In the case n — m, for the function 

(3.7) ei(ar) = e(x) det(x) 
we have 

(3.8) Z-(ei,a - to) = Z + (e,a + 1 - m) = c m r m ((a+ I)/ 2 ), 

7r m2 / 2 
r TO (m/2) 

Formulas (3.6) and (3.8) extend meromorphically to all complex a, excluding a = 

m — 1, to — 2, . . . for (3.6), and a = to — 2, m — 3, . . . for (3.8). Excluded values 

are poles of the corresponding gamma functions written for to > 2. If to = 1, 
these poles proceed with step 2, namely, a = 0, —2, —4, . . . and a = — 1, —3, . . . , 
respectively. 

3.1.2. Analytic continuation. In the following, throughout the paper, we 
assume m > 2. 

Lemma 3.2. Let f e S(9Jt„ iTO ). For flea > m — 1, i/ie integrals (3.2)-(34) are 
absolutely convergent. For Re a < m — 1, they extend as meromorphic functions of 
a with the only poles 

to — 1, to — 2, ... , for (3.2) and (3.3), 

to — 2, to — 3, ... , for (3.4). 

These poles and their orders are exactly the same as those of the gamma functions 
r m (a/2) and r m ((a + l)/2), respectively. The normalized zeta integrals 

Z(f,a-n) Z+(f,a-m) Z-(f,a-m) 

[ ' ' r m (a/2) ' r m (a/2) ' r TO ((o + 1)/2) 

are entire functions of a. 

Proof. This statement is known; see, e.g., [P5], [Kh], [Shi]. We present 
the proof for the sake of completeness. The equalities (3.6) and (3.8) say that the 
functions a — ► Z{f, a — n) and a — ► Z_(f, a — m) have poles at least at the same 
points and of the same order as the gamma functions r m (a/2) and T m ((a + l)/2), 
respectively. Our aim is to show that no other poles occur, and the orders cannot 
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exceed those of T m (a/2) and T m ((a + l)/2). Let us transform (3.2) by passing to 
upper triangular matrices t £ T m according to Lemma 1.12. We have 

(3.10) Z(f, a - n) = / F(* M , . . . , t m , m ) J] *£"'** , 
F(ti,i,...,t m ,m) = / dt* / f(vt)dv, dU = Y[dtij. 

R m(m-i)/2 y njm 1< J 

Since -F extends as an even Schwartz function in each argument, it can be written 

as 

F(ti,i, . . . ,t m , m ) = ^0(^1,1, • • • ,t m ,m)i 

where Fq £ S(R m ) (use, e.g., Lemma 5.4 from [Tr, p. 56]). Replacing t\ i by Si,i, 
we represent (3.10) as a direct product of one-dimensional distributions 

m 

(3.11) Z(f,a-n) = 2- m (H( Siti ) ( ?- i - 1)/2 , F (s M , . . . ,a m , m )), 

i=l 

which is a meromorphic function of a with the poles m— l,ra — 2, . . . , see [GShl]. 
These poles and their orders coincide with those of the gamma function r m (a/2). 
To normalize the function (3.11), following [GShl], we divide it by the product 

m m— 1 

J] T((a i + l)/2) = [] r ((« - 0/2) = r m ( a /2)/rr m (" 1 - 1 )/ 4 . 

i=l i=0 

As a result, we obtain an entire function. 

For the distribution Z-(f, a — m), the argument is almost the same. Namely, 
by Lemma 1.12, the integral (3.4) can be written as 



/m 
*(*l,l,...,t m ,m) II*i!» r<d *M 



where 

(3.12) *(ti,i, •••, *m,m) = ^m,m / dt* I f{vt)sgndet(v)dv. 

R m(m-l)/2 O(m) 

Since <I> extends to M. m as an odd Schwartz function in each argument, then 

1 /" m 

(3.13) £_(/, a - m) = — / *(*i,i> . . . , U,m) [J M a- ' s S n fe,0 *M- 

By the well known theory [GShl, Chapter 1, Section 3.5], this integral extends as 
a meromorphic function of a with the only poles m — 2,m — 3,.... To normalize 
this function, we divide it by the product 

As a result, we get an entire function. □ 
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Analytic continuation of integrals (3.2)-(3.4) can be performed with the aid of 
the corresponding differential operators. In particular, for Z±(f,a — m), one can 
utilize the Cayley differential operator 

(3.14) V = dek(d/dxi,j), x = (x itj ) e Wl m , 

that enjoys the following obvious relations in the Fourier terms: 

(3-15) (?[Df\)(v) = H) m dct(y) (?f)(y), 

(3.16) V{Tf){y) = i m (f[f(x) det(x)])(y). 
Lemma 3.3. Let x G 0Jl m , rank(x) = m. For any AeC, 

(3.17) 2?[|dct(x)| A ] = (A,m)|det(x)| A " 1 sgndet(x), 

(3.18) 2?[|dct(x)| A sgndct(a;)] = (A, m) |det(x)| A_1 . 

Proof. Different proofs of these important formulas can be found in [Ra] and 
[P2]; see also [Tu, p. 114]. All these proofs are very involved. Below we give 
an alternative proof which is elementary. Note that (3.17) and (3.18) follow one 
from another. Furthermore, it suffices to assume that A is not an integer (A ^ Z). 
Once (3.17) and (3.18) are proved for such A, the result for A G Z then follows by 
continuity. 

We start with the formula 

(3.19) V x [f{ax)\ = det(a) (Vf)(ax), a G GL(m,R), 

which can be easily checked by applying the Fourier transform to both sides. Indeed, 
if / is good enough at infinity (otherwise we can multiply / by a smooth cut-off 
function) then by (3.15), the Fourier transform of the left-hand side of (3.19) reads 

Hrdet(y).F[/M](2/) - ( ~^yV /)((0^) 

which coincides with the Fourier transform of the right-hand side. If f(x) = 
|det(x)| A then (3.19) yields 

|dct(a)| A £>|dct(x)| A = dct(a) [V |dct(-)| A ](a:r). 

By setting a = x^ 1 (recall that rank(a;) = m so that x is non-singular) we obtain 

V |det(a;)| A = A|det(a;)| A_1 sgndct(a;), A = [T> |det(a;)| A ](/ m ), 

and therefore 

(3.20) V [|dct(x)| A sgndet(x)] = 4|det(a;)| A-1 . 
In order to evaluate A, we make use of the Gaussian functions 

e(x) = cxp(-tr(x'x)) and ei(x) = e(x) det(x) = (-2) -m (De)(a;), 
see (3.5) and (3.7). By (3.8), 

Z_(ei, A) - Z+(e, A + 1) = c m r m ((A + 1 + m)/2) 



3.1. ZETA INTEGRALS 49 

(here and on we do not care about the poles because we assumed in advance that 
A £ Z). On the other hand, by (3.20) and (3.6), 

Z_(ei,A) = {-2)- m Z_{Ve,X) 

= 2- m (e(x),2?[|dct(x)| A sgndct(:r)]) 
= 2-"MZ + (e,A-l) 
= c m 2-"\4r m ((A-l + m)/2). 
Hence, owing to (1.9), we obtain 

A= 2-T ( (A + l + m)/2) r(l-A) 

r m (A-l + m)/2) v ; r(l-A-m) l ' ; 

D 

Formulas (3.17) and (3.18) imply the following connection between zeta inte- 
grals Z + (-) and Z-(-). 

Corollary 3.4. 

, r(a + 1 — m) 



(3.21) Z^(f,a-m)=c a Z ± CDf,a + l-m), c a = {-l) r 

T(a + 1) 

The formula (3.21) can be used for analytic continuation of the zeta integrals 
Z T (f,a-m). 

3.1.3. Functional equations for the zeta integrals. The following state- 
ment is a core of the theory of zeta integrals (3.2)-(3.4). 

Theorem 3.5. Let f G S(Wl n . rn ), n > m. Then 



r m (a/2) ' - r m ((n-a)/2)' 

,333) Z +(f, a-m) = n -m 2 /2 om(a-m) Z +(F.f,-a) 



r m (a/2) ' ' r m ((m-a)/2)' 



Z_(ff,-a) 



(Q 941 Z -V> a m ) _ (_j\m -m 2 /2 om(a-m) 

1 ' j r m ((a + l)/2) l J r ro ((m-a + l)/2)- 

Proof. We recall that both sides of each equality are understood in the sense 
of analytic continuation and represent entire functions of a. The equalities (3.22) 
and (3.23) were obtained in [Far], [FK], [Ge], [Ra], [P2] in the framework of more 
general considerations. A self-contained proof of them and detailed discussion can 
be found in [Ru7]. The equality (3.24) was implicitly presented in [P2, p. 289]. In 
fact, it follows from (3.23) owing to the formulas (3.21) and (3.15). We have 

Z_ (/, a — ra) c a Z + (Vf, a + 1 — ra) 

r m ((a + l)/2) = r m ((a + l)/2) 

Z+(T[Vfl-a-l) 



r —.— m /2 rjm(a+l— rn) 

r m ((m-a-l)/2) 

(-i\™ r- ^-™V2 om(cH-l-m) (C?7) i^h dct (v) \ det (v) T"' 1 ) 

[ } a r ro ((m-a-l)/2) 

Z_(ff,-a) 



r m ((m-a + l)/2) 



.-,() 
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where (use (3.21) and (1.9)) 



/ ■\m -m 2 /2nm(otl-m) m[\jn a + lj/2) _ , , m - m 2 /2 nm(a-m) 

[ I) C a 7T Z. .. . . . — ^ I) IT A 

lm((m - a - 1/2) 



3.2. Normalized zeta distributions 



□ 



It is convenient to introduce a special notation for the normalized zeta integrals 
(3.9) which are entire functions of a. Let 



(3.25) 



Ca(x) 



r m (a/2)' 



x eWlr, 



n > m, 



(3.26, CW-IO-Wf 



|dct(x)| Q - m sgndet(ai) 
lm((a + l)/2) 



(3.27) (Ca,/) = O.C. 



r m (a/2) 

Given a Schwartz function /, we denote 

/(*)Ca (x) (fc, (Ct , /) = O.C. I /(*)£ (*) ^ 

SW»,m 9K m 

where"a.c." abbreviates analytic continuation. We call Ca and Ca normalized zeta 
distributions of order a on SDT n)Tn and 9Jl m , respectively. By (3.21), 

(3.28) (Ca,/) = Ca(C+l,*V). (C+J) = d a (C- +1 ,Vf), 

where 

r(a + 1 — to) 



[ ' r(a + l) ' 

Ca T m (a/2 + 1) T(a + 1 - to) T(to - a) 



(use (1.9)). 



T m (a/2) 2T(a + l)r(-a) 

Our next goal is to obtain explicit representations of the distributions (3.27) 
when the corresponding integrals are not absolutely convergent. Evaluation of 
(Ca, /) when a is a non-negative integer is of primary importance in view of subse- 
quent application to the Radon transform in Chapters 5 and 6. 



(3.29) 



Theorem 3.6. Let f e S(97l n , m ). Fora = k, k = 1, 2 

^(n— fc)m/2 



(Cfe,/) = 



r m (n/2) 



f/7 



Furthermore, in the case a = we have 

jiim/2 



(3.30) 



(Co,/) 



/ 7 



/(0). 



dw. 



r m (n/2)' 

Proof. STEP 1. Let first fc > m — 1 . In polar coordinates we have 
1 



(a,/) = 



r m (fc/2) 



9K„ 



r m (fc/2) 



f(x)\x\*- n dx 



\r\ k l 2 - d dr j(l 



r l/2 





cfy. 



O(n) 
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(3 
I n -k 
replace the integration over 0(k) by that over Vk, m - We get 



Now we replace 7 by 7 



j3 € O(k), then integrate in [3 G O(k), and 



(&,/) 



ffc, m r m (fc/2) 

O(n) P m 

(set w = vr 1/2 e 9Jlfe, m ) 



dl I \r\ k/2 - d dr / /(7 
v k , m 



vr 1 / 2 




rfr 



ffc, m r m (fc/2) 



an fc 



dw y / ( - 

O(n) 



U) 





cfy. 



This coincides with (3.29). 

STEP 2. Our next task is to prove that analytic continuation of (£ a , /) at the 
point a = k (< m — 1) has the form (3.29). To this end, we express £ a through the 
Garding-Gindikin distribution; see Section 2.3. For Re a > m — 1, by passing to 
polar coordinates, we have (£ Q , /) = 2~" l <7„. m t/ Q /2(i 7 '), where 



0a/2(iO = 



r m (a/2) 



F(r)\r 



a/2-d 



dr. 



F(r) 



f(vr 1/2 )dv. 



To continue the proof, we need the following 



Lemma 3.7. Let §('P m ) be the space of restrictions onto V m of the Schwartz 
functions on S m Z> V m with the induced topology. The map 

HTm) 3 F -► /(») = F^s) 

is an isomorphism o/S("P m ) onto i/ie space §(DJl n , m p of 0(n) left-invariant func- 
tions on 0Jl ntm . 

This important statement, which is well known for m = 1 (see, e.g., Lemma 
5.4 in [Tr, p. 56]), was presented in a slightly different form by J. Farau [Far, 
Prop. 3] and derived from the more general result of G. W. Schwarz [Sc, Theorem 
1]. According to (2.28), analytic continuation of Q a /2(F) at the point a = k, 
k = 1, 2, . . . , m — 1, is evaluated as follows: 

Qk/2{F) = n- km / 2 J F(w'u)duj 

m k . m 



-km/2 



du> 



/ 7 



{uo'uo) 1 / 2 




d~f. 



£V„, 



Wl k , m 0(n) 

By making use of the polar coordinates, one can write u>' G 9Jl m .k as 

J = (3u (wJ) 1/2 , f3eO(m), u Q 
Hence, u> = (ww') 1 / 2 ^/?' , and 

{Jujfl 2 = {(3u ujJu' f3') 1/2 = /3m (wu;') 1/2 Uo/3' = f3u 0J. 



Ik 





-,2 
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P 



By changing variable 7^7 

Gk/2(F) = lT- km ' 2 
„-km/2 



, we obtain 



Wl k 



dw / / ( 7 

O(n) 



(3uoLU 





^7 



$H fc 



rf^ J / ( 

O(n) 







dry, 



and (3.29) follows. For a = 0, owing to (2.27), we have 

nm/2 

(CoJ) = 2-^, ro F(0) = f - w ^/ ( ) . 



n 



Remark 3.8. For n > m, the integration in (3.29) over 0(n) can be replaced 
by that over SO(n). 

The following formulas for (£&, /) are consequences of (3.29). We denote 

7r (nm—km—nk)/2 p (n/2) ~-(m—k)(n/2—k) 



(3 ' 31) Cl ~ 2*F m (n/2) ' 

Corollary 3.9. For all k = 1,2,..., n, 



('2 



r fc (fc/2)r m _ fc ((n-fc)/2)- 



(3.32) 

Moreover, if k = 1,2, ... ,m — 1, then 



(Cfc>/) = ci dv f(vu)dw. 



(3.33) 

(3.34) 



(Cfe,/) = Cl I du I f{yu')\y\l!- n dy 



('2 



v m , k 


m n , k 




1 

m n . k 


dy f 

\y\T m J 
m k , m - t . 


f([y;yz])dz. 



Proof. From (3.29) we have 



^(n— fc)m/2 

(Cfc,/) = f; — /n , I duj I /(7A w)c?7 I A = 



r m (n/2) 



h 




€ V n ,k 



M k , m 0(n) 
7 ^(n—k)m/2 



o- n ,kT m (n/2) 



dio / f(vuj)dv 



Wl k . m v n ,„ 



3.2. NORMALIZED ZETA DISTRIBUTIONS 53 

which coincides with (3.32). To prove (3.33), we pass to polar coordinates in (3.32) 
by setting u/ = ur 1 / 2 , u e V myk , r ^V k . This gives 

(C fc ,/) = 2- fc Cl J dv J \r\( m - k -VI 2 dr J f(vr^ 2 u')du 
v n , k v k v m , k 

= Cl J du J f(yu')\y\l!- n dy. 

v ^ wi n , k 

To prove (3.34), we represent u> in (3.32) in the block form u = [n; £], 77 G SDtfe.fe, C £ 
9Kfc,m-fc, and change the variable Q = r\z. This gives 

(&,/) = ci / |J7| m - fc d^ / dz f f(v[ mV z])dv. 

m k , k m k , m - k Vn.k 

Using Lemma 1.11 repeatedly, and changing variables, we obtain 

(C fc ,/) = 2~ k c 1 a k , k J\r\( m - k - 1 V 2 dr J dz J f{v[r 1 / 2 ,r 1 l 2 z])dv 
v * tOl h , m . k v n , k 

= c x a k ,k I 1 ,„_ TO / /([y;^])^- 

97l„, fc 9Tt fc , m _* 

By (3.31), (1.37) and (1.10), this coincides with (3.34). □ 

The representation (3.34) was obtained in [Shi] and [Kh] in a different way. 

Remark 3.10. Another proof of Theorem 3.6, which does not utilize Faraut's 
Lemma 3.7, was given in [Ru7]. The proof given above mimics the classical ar- 
gument (cf. [GShl, Chapter 1, Section 3.9] in the rank-one case) and is very 
instructive. 

One can also write ((&, /) as 



(3.35) (<*,/) = / f{x)du k {x), /6S(97t„, m ), 

Wl n , m 
where v k is a positive locally finite measure v k defined by 



(3.36) (v k ,<p)=ci dv <p(vu))du>, »j £ C c (9JI„, ra ), 

v ^ m k<m 

C c (9H„ ]m ) being the space of compactly supported continuous functions on Wl n , m : 
cf. (3.32). In order to characterize the support of v k , we recall that 971^ ' m stands 
for the submanifold of matrices x G 97tn,m having rank k, and denote 

k 

(3.37) m { n % = |J m^ m (the closure of M^J. 

j=o 

Lemma 3.11. The following statements hold. 
(i) supp^ fc = 9Jt . 



.-,i 
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4 o 





under the group G of 

: m 

g 2 eGL(m,R), 



(ii) The manifold < $fl) l ' m is an orbit of ek = 
transformations 

x — ► gixg2, 51 £ GL(n,l 

and 
(3.38) dimSDt^ = fc(n + m-fe). 

(iii) The manifold 2H„ m is a collection of all matrices x G VJl n , m of the form 



(3.39) 

or 

(3.40) 



.t = 7 







X = VUJ, 



7 G 0(n), us effl k 



v e K./t, w e SDTfc.r, 



Proof, (i) Let us consider (3.36). Since r&nk(voj) < k, then (i/^, ^?) = for all 

(m (m 

functions tp G G c ($H„ !m ) supported away from 9Jt nm , i.e. suppz/fc = 9Jt n m- 

(ii) We have to show that each x G 9Jt„ ^ is represented as x = g\ekgi for 
some gi G GL(n, R) and 92 £ GL(m, R). We write x = us, where u G V^ >m and 
s = (x'x) 1 ' 2 is a positive semi-definite m x m matrix of rank fc (see Appendix C, 

4 



11). By taking into account that s = g' 2 







52 for some gi G GL(rn, 



and u = 7 



J 7/ 



for some 7 G O(n), we obtain x = gxe^g^ with 



.91 =7 



.92 

U -Ln—m 



G GL(n,R). 



Therefore, the manifold 971^ ^ is a homogeneous space of the group G. This allows 
us to find the dimension of dK n ' m by the formula 



dim an: 



(k) 



dimG — dimGi, 



where G\ is the subgroup of G leaving e^ stable. In order to calculate the dimension 
of Gi, we write the condition gxe^gi = e k in the form 



Oi 


«2 




4 
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&2 




4 





a 3 


a 4 












&3 


&4 











where 01 G GL(fc,R) and 61 G GL(fc,R). This gives 01 61 = 4, 03 = 0, fo 2 = 0. 
Hence, 

dimGi=n — k(n — k) + m(m — k), 

and (3.38) follows. 

(iii) It is clear that each matrix of the form (3.39) or (3.40) has rank < k. 



Conversely, if rank(x) < k then, as above, x = us = 7 



where 7 G 0(n) and 



s = (x'x) 1 ' 2 is a positive semi-definite m x m matrix of rank < k. The latter can 
be written as 



g\g', geO(m), A = diag(Ai, . . . , A fe ,0, . . . ,0), 
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"> 



and therefore, 



x = 7 



<7A 





.9=7 



5 

U ^n—m 



g =7i 







.9 

U 1-n—m 



and w G ®tfe,m- The representation (3.40) follows from 



where 71 = 7 

(3.39). D 

Corollary 3.12. The integral (3.35) can be written as 

(3.41) (Cfc,/)= / f{x)dv k {x) = J f(x)du k (x). 

rank(a:)< k rank(a:) — k 

Proof. The first equality follows from Lemma 3.11 (i). The second one is 
clear from the observation that if rank(x) < k — 1 then, by (3.40), x = vu>, v G 
Vn.k-i, w € 9Jl/;_i iTO . The set of all such pairs (v,w) has measure in V n ,k x 

anib, m . □ 

Remark 3.13. We cannot obtain simple explicit representation of the conjugate 
normalized zeta distributions (C k , /), k — 0, 1, . . . , m — 1. At the first glance, it 
would be natural to use the formula 

(c,/) = c a (£ +1 ,vf), Ca = (-ir r( " + 1 ~, m) , 

1 (a + 1) 

see (3.28), in which (Ca+i>^/) can ^e evaluated for a = k by Theorem 3.6 or 
Corollary 3.9. Unfortunately, we cannot do this because c a — 00 for such a . On 
the other hand, Lemma 3.2 says that (( k , f) is well defined by (3.13), namely, 

(3.42) (C fe -,/) = 2-"V- m ( m - 1 )/ 4 (cc fc ,$), 
where 

|i M | a - 4 sgn(i M ) 



^>k\tlS, • ■ • 7 t m . m ) 
$(<1,1, • ■ • , t m ,m) 

In particular, for fc = 0, 



n 



LT((a- *)/2 + l) 



d£* / /(wt)sgndct(w) cfo. 

i(m-l)/2 ( m ) 



(3.43) 



wo(£i,i, • • • i t m ,m) 11 



\ r(A/2 + i) 



A=-i 



where the generalized functions 
|s| A sgn(s) 



L(A/2 + l) 

arc defined as follows. For i odd: 

( \s\ x sgn{s) 

{ r(A/2 + i) 



A=-j 



A=-i 



r(i-»/2) 



i = 1,2,. ..,m, 



(*"*,¥>) 



,s- 4 p(«) - p(-«) - 2 



V(o) + | [ ^"(o) + ... + ^^ I ^- 2 )(o) 



(is . 
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For i even: 

( k| A sgn(s) 
^(A/2 + l) 



\ (— l)*/ 2 ^-^(O) (t/2 — 1)! 

*=v*r f^ 



sec [GShl, Chapter 1, Section 3.5] . Note that the Fourier transform of the distri- 
bution C^~ has the form 

(—i) m tt™ 2 / 2 

(3.44) ^o-)(l/) = r ^ (( i n + 1)/2) Bgndet( g ). 

This follows immediately from (3.24) and the Parseval formula (1.5). 

In the following the expression (£q~ , /) will be understood in the sense of regu- 
larization according to (3.42), (3.43). 

3.3. Riesz potentials and the generalized Hilbert transform 

The functional equation (3.22) for the zeta distribution can be written in the 
form 

(3.45) Vt^(/> a - n) = (2^)- nm Z{Tf, -a) , 

7n,m(a) 

2 am j«m/2 T m (a/2) 

(3.46) 7 (a) = — — — - ^— t — , a^n-m + 1, n-m + 2, .... 

r TO ((n-a)/2) 

We recall that m > 2. The excluded values a = n — m+1, n — m + 2, ... are 
poles of the gamma function T m ((n — a)/2). Note that for n = m, all a = 1, 2, . . . 
are excluded. The normalization in (3.45) gives rise to the Riesz distribution h a 
defined by 

(3.47) (h a ,f) = 2-« m 7r-™"/2 rm ^l_^ (Cq)/) 

- a-c. -^— f \x\ a m n f{x)dx, 

ln,m{a) J 

Wln,m 

where / G §(9Jl„ !m ) and "a.c." abbreviates analytic continuation in the a-variable. 
For Re a > m — 1, the distribution h a is regular and agrees with the usual function 
h a (x) = |ic|m _ "/7n m{ a )- The Riesz potential of a function f G §(SDt„ TO ) is defined 
by 

(3.48) (I a f)(x) = (h a ,f x ), f x {.) = f(x - ■). 

For Re a > m — 1, one can represent /"/ in the classical form by the absolutely 
convergent integral 

(3.49) (/«/)(£) = ^_ /" /(a;-y)|y|«-"dy. 

7n,m(a) 7 

We also introduce i/ie generalized Hilbert transform 

(3-50) (W/)(aO = (Co - ,/*) = (Co" */)(*)• 

By (3.44), this can be regarded as a pseudo-differential operator with the symbol 

(— i\ m it™ 2 / 2 
(3.51) ^o)(y) = J }J m+1)/2) s g n d et(y). 
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Clearly, H extends as a linear bounded operator on L 2 (Tl n ,m)- For m = 1, it 
coincides (up to a constant multiple) with the usual Hilbcrt transform on the real 
line. 

The following properties of Riesz potentials and Riesz distributions are inher- 
ited from those for the normalized zeta integrals. We denote 

(3.52) 7l = 2-*™7r- fe ™/ 2 r ro (^)/r ro g 

(3.53) 72 = 2 - fe ( m+1 ) 7r -fe(™+")/ 2 r fe ( !L ^)- 

Theorem 3.14. Let f G S($H„. m ) 7 n > m. Suppose that a = k is a positive 
integer. If k =/= n — ro + 1, n — m + 2, . . ., i/ien 



(3.54) (7 fc /)(x) = 71 y du; J /(,- - 







cfy, 



(3.55) = j2 dv / / (x — vu>)du>. 

v ^» M k , m 
Furthermore, 

(3.56) (I°f)(x) = /(*). 

This statement is an immediate consequence of Theorem 3.6. 

Lemma 3.15. Let f 6 S(9D r t TJ . ;Tri ), n > m, aeC, q^ n — m+1, n — m + 2, .... 
(i) TTie Fourier transform of the Riesz distribution h a is evaluated by the formula 
(J r h a )(y) = |y|m Q 7 ^ e precise meaning of which is 

(3.57) (h a J) = (27r)- nm (\y\ m a , [Tf){y)) = (2ir)- nm Z(ff, -a). 
(ii) If k = 0,1, ... , and A is t/ie Cayley- Laplace operator, then 

(3.58) ft a = (-l) mk A k h a+2k , U. (h a ,f) = (-l) mk (h a+ 2 k ,A k f). 
(iii) 7/n = m, k = 1, 2, . . . , and I? is i/ie Cayley operator, then 

(3.59) ^ = c© 2fe - 1 C +2fe _ 1 , i.e. (h a ,f) = c(-l) m ((- +2k _ 1 ,V 2k - 1 f), 

_ (-1)^(^+1) r m (l + (m - a - 2fc)/2) 

C ~~ 2( Q + 2fe_1 ) m 7T m2 / 2 ' 

Proof, (i) follows immediately from the definition (3.47) and the functional 
equation (3.45). To prove (3.58), for sufficiently large a, according to (1.49), we 
have 

A k h a+2k (x) = ±^A k \xC +2k ~ n 

7n,m(a + 2fc) 



7 „, m (a + 2fc)' lm 
= ch a (x), 
where by (1.50) and (1.9), 

_ Bk(a)in,m(a) _ B k (a) T m (a/2) T m ((n - a)/2 - k) _ ( _ , mk 
C ~ 7n, m (a + 2fc) " 4™ fc r m (a/2 + fc)r m ((n-a)/2) ~ l ' 
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For all admissible a G C, (3.58) follows by analytic continuation. Let us prove (iii). 
Owing to (3.58), h a = {-l) mk V 2k - 1 Vh a+2k - Since, by (3.47 ) and (3.28), 

_ r m ((m-a-2fc)/2) + + _ T(a + 2k) 

n a+2k 2( a + 2k ) m n m2 / 2 ^"+2fc> u ^a+2k T(a + 2k - m) a+2k ~ 1 ' 

then (3.59) follows after simple calculation using (1.9). □ 

Lemma 3.15 implies the following. 

Theorem 3.16. Let f e S(9Jt„, m ), n>m. 
(i) If k = 0,1,2,..., then 

(3.60) (I- 2k f)(x) = (-l) mk (A k f)(x). 
(ii) If k = 1,2,..., and n > m, then 

(3.61) (J 1 - 2fc /)(a;) = (-l) mk {I 1 A k f)(x) = Ci f dv f (A k f){x - vy') dy, 

S n-1 r 

_ (-l) mfe F((n-TO)/2) 

(iii) If k = 1,2,..., and n = m, then 

ri-2fc n^ _ „ rt/rtfk-i ^ - - (-l) m(fe+1) r m ((m + l)/2) 



(3.62) {l'- 2k f){x)=c 2 (HV 2k -\f){x), c 2 

7i fremg i/ie generalized Hilbert transform (3.50). 



1/2 



PROOF. The equality (3.60) is a consequence of (3.48), (3.57), and (3.58). 
Namely, 

(r 2k f)(x) = (h_ 2k ,f x ) {3 ^ } (-ir k (h ,A k f x ) 

(3 = 7) {-l) mk (2ir)- nm Z(T(A k f x ), 0) 

= (-l) mk (A k f x )(0) = (-l) mk (A k f){x). 
Similarly, by (3.48), {P~ 2k f){x) = (fci_ 2 fc,/*). If n > m, then, by (3.58), 

(fti- 2fc ,/x) = (-ir fc (/n, A fc /,) = (-ir*(/n, (A fe /),) = (-lr'^AV)^), 

and it remains to apply (3.55) (with k = 1). If n = to, then we apply (3.59 ) with 
a = 1 — 2k and get 

(hi-2k,U) = (-l) m c 2 (Co-,© 2 *- 1 ^) = C2 (Co", (Z> 2fc "7)x) = c 2 (HV 2k - l f){x). 

D 

Integral representations (3.49) and (3.54) (or (3.55)) can serve as definitions of 
I a f for functions / belonging to Lebesgue spaces and a belonging to the associated 
Wallach-like set 

(3.63) "W n ,m — {0, 1, 2, . . . , fco} U {a : Rea>m—1; a =/= n — to+ 1, n — m + 2, . . .}, 

fco = min(m — 1, n — to). 
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Theorem 3.17. [Ru7]. Let f £ L p (9Jt„, m ), n > to. If a e W„, TO , then the 
Riesz potential (I a f)(x) absolutely converges for almost all x £ 9Jt n ,m provided 

n 

(3.64) 1<P<-^ -• 

He a + to — 1 

For to = 1, the condition (3.64) is well known [Stl] and best possible. We do 
not know whether (3.64) can be improved if to > 1. 

3.3.1. Inversion of Riesz potentials. Let us discuss the following problem. 
Given a Riesz potential g = I a f, how do we recover its density /? In the rank-one 
case, a variety of pointwise inversion formulas for Riesz potentials is available in 
a large scale of function spaces [Rul], [Sa]. However, in the higher rank case we 
encounter essential difficulties. Below we show how the unknown function / can be 
recovered in the framework of the theory of distributions. 

First we specify the space of test functions. The Fourier transform formula 

(h a ,f) = (2n)- nm (\y\ m a ,(Ff)(y)) 

reveals that the Schwartz class § = S(9Jt„. m ) is not good enough because it is not 
invariant under multiplication by j/|„ Q . To get around this difficulty, we follow 
the idea of V.I. Semyanistyi [Se] suggested for m = 1. Let \P = ^>(Tl n , m ) be the 
subspacc of functions tp(y) £ § vanishing on the set of all matrices of rank < to, 
i.e., 

(3.65) {y : y £ 9Jt„, m , rank(y) < to} = {y : y € 93t n , m , \y'y\ = 0}, 

with all derivatives. The coincidence of both sets in (3.65) is clear because rank(y) = 
rank(y'y), see, e.g., [FZ, p. 5]. The set *& is a closed linear subspace of S. Therefore, 
it can be regarded as a linear topological space with the induced topology of §. Let 
<f> = <fr(9Jt„ ]rn ) be the Fourier image of \I/. Since the Fourier transform T is an 
automorphism of S (i.e., a topological isomorphism of § onto itself), then <J> is 
a closed linear subspace of S. Having been equipped with the induced topology 
of S, the space $ becomes a linear topological space isomorphic to vp under the 
Fourier transform. We denote by $' = $'(9DT n)Tn ) the space of all linear continuous 
functionals (generalized functions) on <I>. Since for any complex a, multiplication 
by |j/|^, a is an automorphism of \&, then, according to the general theory [GSh2], 
I a , as a convolution with h a , is an automorphism of $, and we have 

Wfliv) = \y\ m a (Ff)(y) 

for all ^'-distributions /. In the rank-one case, the spaces $, vf, their duals and 
generalizations were studied by P.I. Lizorkin, S.G. Samko and others in view of ap- 
plications to the theory of function spaces and fractional calculus; see [Sa], [SKM], 
[Rul] and references therein. 

Theorem 3.18. Let a £ W„ jm and let f be a locally integrable function such 
that g(x) = (I a f)(x) is well defined as an absolutely convergent integral for almost 
all x £ $H„ im . Then f can be recovered from g in the sense of <!>' -distributions by 
the formula 

(3.66) (f,<j>) = (g,I- a <j>), </>£$, 
where 

(3.67) (/-»(*) = (T-^y^T^ix). 
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In particular, if a = 2k is even, then 

(3.68) (/,<}>) = (-l) mk (g,A k cj>), e *, 

A being the Cayley- Laplace operator (1.44)- 



CHAPTER 4 



Radon transforms 



4.1. Matrix planes 

Definition 4.1. Let k,n, and m be positive integers, < k < n, V n ^ n _k be 
the Stiefel manifold of orthonormal (n — fc)-frames in R", 9Jt„_fc im be the space of 
(n — k) x m real matrices. For £ £ Ki,n-fe an d £ £ 9Kn-fc,m, the set 

(4.1) r = r(£, t) = {x : x g 9Jt„, m ; £'a: = t} 

will be called a matrix k-plane in 9Jl„ iTO . For fc = n — m, the plane t will be called 
a matrix hyperplane. We denote by % the manifold of all matrix k- planes. 

The parameterization r = t(£,£) by the points (£,£) of a "matrix cylinder" 
is not one-to-one because for any orthogonal transformation 9 £ 



n—k 



xWl 



n—k,m 



0(n — k), the pairs (£, t) and {(,&', Ot) define the same plane t. We identify functions 
<p(t) on T with the corresponding functions <p{£,t) on V n ^ n -k x 9Jt n _fc jTO satisfying 
<p(£6' , 9t) = cp(£, t) for all g 0(n — k), and supply X with the measure dr so that 



(4.2) 



ip(r) dr = 
T y„.„_ fc x97l„ 



(/?(£, t) d£di. 



The plane r is, in fact, a usual fcm-dimcnsional plane in 



To see this, we 



write x — (xij) G 9Jt„. m and t = (ti 





/ a;i,i \ 


(4.3) x = 


a?i,2 


\ %n : m / 


and denote 


(4.4) £ = 


Then (4.1) reads 


(4.5) 


r = 



g 2ttn-fc 

i = 



as column vectors 

\ 

g 



*i,i 

*1,2 



(n—k)m 



diag(C, 



\ v<n— k,m / 
) S/ ^ Mim,(n— fe) 



r(£,i) = {x:xeR" m ; £'z = t}. 

The fcm-dimcnsional planes (4.5) form a subset of measure zero in the affine Grass- 
mann manifold of all fern-dimensional planes in R" m . 

The manifold X can be regarded as a fibre bundle, the base of which is the 
ordinary Grassmann manifold G n ,k of fc-dimcnsional linear subspaces of R™, and 
whose fibres are homeomorphic to 0Jl n -k, m - Indeed, let 

7T : X — ► G ni k, 

be the canonical projection which assigns to each matrix plane r(£, t) the subspace 
(4.6) n = v(0 = {y:yeR n ; ?y = 0} g G n . k . 
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The fiber H v = ir^ 1 ^) is the set of all matrix planes (4.1), where t sweeps the 
space dyin~k.m- Regarding T as a fibre bundle, one can utilize a parameterization 
which is alternative to (4.1) and one-to-one. Namely, let 

(4.7) x=[xi...x m ], t=[t 1 ...t m ], 

where Xi G R™ and ti G M. n ~ k are column- vectors. For r = r(£, t) G T, we have 

t = {x -. x e Tl n ^ m ; i'xi = u, i = l,..., m}. 

Each ordinary /c-plane r» = {xi : Xi G R"; £'xj = ti} can be parameterized by the 
pair (77, Aj), where r\ is the subspace (4.6), Aj G r/" 1 ", i = 1, . . . ,m, are columns of 
the matrix A = ££ G $H„. m , and 77- 1 denotes the orthogonal complement of r\ in R™. 
The corresponding parameterization 

(4.8) t = t(?7,A), r]EG n , k , A=[Ai...A m ], A t e t/ 1 , 

is one-to-one. Both parameterizations (4.1) and (4.8) will be useful in the following. 



4.2. Definition and elementary properties of the Radon transform 

The Radon transform / of a function f(x) on $H„. m assigns to / a collection 
of integrals of / over all matrix fc-planes (4.1). Namely, 



f(r) 



./'(■' 



rel. 



x£t 



In order to give this integral precise meaning, we note that the matrix plane r 
t(£,£), £ G K.n-fe, t G $H„__ fe , m , consists of "points" 



<)(, 



t 



where u> G 9Hfe !m , and g^ G SO(n) is a rotation satisfying 



(4.9) 



g& = £, £0 





ln—k 



g y„,„- 



fe- 



This observation leads to the following 



Definition 4.2. The Radon transform of a function f(x) on Wl n , m is defined 
as a function on the "matrix cylinder" V n , n -k x 9ftn-fc,m by the formula 



(4.10) 



f(r) = m,t) 



f U? 






dw. 



$H fc 



It is worth noting that the expression (4.10) is independent of the choice of the 
rotation g^ : £ ~ > £■ Indeed, if <?! and 52 are two such rotations, then gi = g 2 g 
where g belongs to the isotropy subgroup of £ - Hence g has the form 



9 



9 

I n -k 



9 G SO(k). 
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Multiplying matrices and changing variable 8lo — ► to, we get 



/ [91 



U! 
t 



cLu = 



Tl k 



m k 



f [92 

f (92 





I n -k 



us 

t 



du> 



Wl,., 



t 



du). 



Furthermore, since 

(4.11) 9i 

one can write 
(4.12) 

(4.13) 



u 




U! 




' " 




LU 


t 


= 9t 





+ .9S 


t 


= 9t 






•#, 



m,t) 



f\9t 



to 




an*., 



-£t\duj 



{yeffl„, m ■. Cv=o} 



If m — 1, then /(£,£) is the ordinary fc-plane Radon transform that assigns to 
a function f(x) on R™ the collection of integrals of / over all ^-dimensional planes. 
If m = 1 and fc = n — 1, the definition (4.12) gives the classical hyperplane Radon 
transform [H], [GGV]. 

In terms of the one-to-one parameterization (4.8), where r = t(t), A), 7/ G 
G„.fe, A = [Ai . . . A TO ] G DJl nyTn , and Aj G r] , the Radon transform (4.10) can be 
written as 



(4.14) 



f(r) 



dyi 



f{[yi + Ai . . . y m + A m ]) dy n 



For m = 1, this is the well known form of the fc-plane transform in R"; cf. [H, p. 
30, formula (56)]. 

The following sufficient conditions of the existence of the Radon transform / 
immediately follow from Definition 4.2. More subtle results will be presented in 
Section 4.7.2. 

Lemma 4.3. 
(i) If f G L (9Jt„, iTO ) ; then the Radon transform f{£,,t) exists for all £ G V n ,n-k an d 
almost all t G 9J?n-fc m - Furthermore, 



(4.15) 



f(Z,t)dt 



f(x)dx, v^y ni „_ t . 



an»_* 



9K„ 



(ii) Let \\x\\ = (tr(x'x)) 1 / 2 = {x\ 1 + . . . + x 2 n m) 1 ' 2 . If f is a continuous function 
satisfying 

(4.16) f(x) = 0(\\x\\~ a ), a>km, 

then /(£,£) exists for all £ G V n ,n-k &nd all t G 9Jl„_fe jTO . 



()! 
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Proof, (i) is a consequence of the Fubini theorem: 



m,t)dt 



Tl n 



Tt n - k , m M k , m 



t 



dio 



Wlr, 



f(g^x) dx = / f(x) dx. 

arr„ 



(ii) becomes obvious from (4.5) if we regard r = r(£, t) as a fcm-dimcnsional plane 

inR" m . D 

Lemma 4.4. Suppose that the Radon transform, 

f(x) — ► /(£,*), x e M, hm , (£,i) g V n . n -k x m n - k . m , 

exists (at least almost everywhere) . Then 
(i) /(£,£) is a "matrix-even" function, i.e., 

(4.i7) K&,to) = f(M), VeeO(n-k). 

(ii) The Radon transform commutes with the group M(n, m) of matrix motions. 
Specifically, if g(x) — jxf3 + b where 7 G 0(n), (3 G 0(m), and b G 9Jl n . m , then 

(4.18) (.f°.9) A (^) = /(7^/3 + £V&). 

Proof, (i) Formula (4.17) is a matrix analog of the "evenness property" of 
the classical Radon transform (the case m = 1, k = n — 1): /(— £, —t) = /(£, t) [H, 
p. 3]. By (4.10), 

m k . m 



where one can choose g^gi = g^ 
9Z8' 



h 
& 



Hence 



et 



m 



h 

6' 



u> 

et 



9i 



t 



which gives (4.17). 
(ii) By (4.10), 



(f°9) A (U) = J /(-(,,, 

M km 



f 9it 



M k 



uj/3 
t/3 

u> 

tp 



+ b I du> (top — ► w) 



.9 7C & ) ) dw 



We set o ^6 = 



p G 3Jlfc im , <? G 97l n _fe,m, and change the variable uj+p 



Then the last integral reads 



m k 



f\9ii 



u> 
tp + q 



dio. 



By (4.9), q = ? g^b = ^ib = (g^oYl'b = £' 7 '6, and we are done. 
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□ 



Corollary 4.5. For x,y G 9K n ,m, let f x (y) = f(x + y). Then 

(4.19) fx(Z,t) = f(S,?x + t). 

Corollary 4.6. If the function f : 9Jt„ !m — ► C is 0(«) left-invariant, i.e. 
f("fx) = f{x) for all 7 £ 0{n), then 



(4.20) 



jW, ft) = /(£, t), V 7 e 0(n), ^O(n-t). 



4.3. Interrelation between the Radon transform and the Fourier 

transform 



We recall that the Fourier transform of a function / G L 1 (9Jl Il!m ) is defined by 



(4.21) 



(ff)(y) = 



exp(i tr(j/'x)) /(a;) da;, y G 9tt„, m . 



9tt„ 



The following statement is a matrix generalization of the so-called projection- slice 
theorem. It links together the Fourier transform (4.21) and the Radon transform 
(4.10). 

For y = [j/i . . . y m ] G 9Jt n , m , let £(y) = lin(j/i, . . . , y m ) be the linear hull of the 
vectors j/i, . . . , y m , i.e., the smallest linear subspace containing j/i, . . . , y m . Suppose 
that rank(y) = £. Then dim£(y) = £ < m. 

Theorem 4.7. Let f e L l (^R. n _ rn ), 1 < k < n - m. If y € 9ft„ iTO; and C 
is a (ra — k)- dimensional plane containing C{y), then for any orthonormal frame 
£ € V n ,n-k spanning £, i/iere exists b G 97l rl _fc,m swc/i £/ia£ y = £o. In £/ns case, 



(4.22) 



TOO/) = 



exp(itr(b't))f(£,t)dt, 



ML-* 



(4.23) 



(•?7)(#0 - ^[/(£, OIW. € G K.,n-fc, & € SOTn-fe.r 



Proof. Since each vector j/j (J = 1, . . . , m) lies in t, it decomposes as j/j = S,bj 
for some 6j G R"~ fe . Hence y = £b where b = [bi . . . b m ] G 9Jt„_fc. m . Thus it remains 
to prove (4.23). By (4.10), 



Ifar = 



Hi 



nm-)](b)= J exp 


(itr(b't))dt 


1 


fU 




w„-,, m m„, m 




LO 

t 


, then, by (4.9), 






£' x = £o5?5£ 


LO 
t 


= ?o 


LO 
t 


= *, 





dw. 



and the Fubini theorem yields 

n/(£, •)](&) - / cxp( i tr(6'e'x)) /(*) da; = (^/)(^6). 
97L. ro 



□ 
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Remark 4.8. It is clear that £ and b in the basic equality (4.22) are not uniquely 
defined. If rank(y) = m one can choose some £ and b as follows. By taking into 
account that n — k > m, we set 



£o = 





In-k 



G K.n- 



W 





i- rn 



GK 



n—k,mi 



Vq 





J 77; 



G14 



so that ^o w o — w o- Consider the polar decomposition 

y = vr 1/2 , v G K.n-fe, r = y'y €V m , 

and let <?„ be a rotation with the property gv^o = v. Then 

y = wr 1/2 = g v v r 1/2 = 0„£oWor 1/2 = £&, 

where 

(4.24) £ = #„£ G K.n-fe, & = w r 1/2 G M n - k , m . 

Theorem 4.9. 
(i) If 1 < k < n — m, then the Radon transform f — ► / is injective on the Schwartz 
space S(9Jt„. m ), and / can 6e recovered by the formula 
2 -m 



/(*) = 



(27T) 



" dr 



(4.25) 



exp 



(-» tar(a/vr 1 / 2 ))(F/(5^o, -))(^or 1/2 )d V . 



(ii) for k > n — m, the Radon transform is non-injective on S(27t n . m ). 

PROOF. By Theorem 4.7, given the Radon transform / of / e S(SDt n , m ), the 
Fourier transform (Tf)(y) can be recovered at each point y € 9Jl n ,m by the formula 
(4.22), so that if / = then Tf = 0. Since T is injective, then / = 0, and we 
are done. Remark 4.8 allows us to reconstruct / from /, because (4.24) expresses 
£ and b through y £ ^Jl n ,m explicitly. This gives (4.25). 

To prove (ii), let ip ^ be a Schwartz function, the Fourier transform of which 
is supported in the set 2ft„ „ of matrices x € %R n ,m of rank m. This is an open set 
in m n , m . By (4.23), 

(4.26) ?$% •)](&) =^b) = V^Vi. V6GOT n _ fe , m , 

because £6 ^ 9^„m (since n — k < m, then rank(£6) < m). By injectivity of the 

Fourier transform in (4.26), we obtain tp(£,t) — V£,£. Thus for fc > n — m, the 
injectivity of the Radon transform fails. □ 

4.4. The dual Radon transform 

Definition 4.10. Let r = r(£,£) be a matrix plane (4.1), (£,£) e V n ,n-k x 
9^n-fe,m- We say that the plane t = t(£, £) contains a "point" x G 9K njm if £'a; = £. 
The dual Radon transform <p(x) assigns to a function <p(r) on 1 the integral of ip 
over all matrix fe-planes containing x. Namely, 

ip(x) = / <p(t), x e ^H n ,m- 

tBx 
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07 



The precise meaning of this integral is 

(4.27) q>{x) = —^— I <p{Z,?x)d£ 



Cn,n—k 



V n , n - k 





ln—k 



£V n 



SO(n) 



Clearly, (p(x) exists for all x G 9Jl n .m if <p is a continuous function. Later we 
shall prove that <p(x) is finite a.e. on 9Jt„ im for any locally integrablc function (p. 

Remark 4.11. The definition (4.27) is independent of the parameterization 
t = r(£, i) in the sense that for any other parameterization r = t(£0',0£), 9 G 
0(n — k) (see Section 4.1), the equality (4.27) gives the same result: 



1 



&n,n — k 



ipW,o?x)dt = 



i 



&n,n—k 



<p(€i,€[%)d£i = &( x )- 



v n , n - k 
Lemma 4.12. The duality relation 



v n . n - k 



(4.28) 



an» 



f(x)Cp(x)dx = 



f(x)(p(x)dx = 



1 



&n,n—k 



dti 



tp(Z,t)f(£,t)dt 



Vn,n-k T\ n 



&n,n—k 



<p(T)f(T) dr 



Wl„ 



is valid provided that either side of this equality is finite for f and if replaced by \f\ 
and \ip\, respectively. 



(4.29) 



PROOF. By (4.10), the right-hand side of (4.28) equals 
1 



&n,n — k 



dt 



"»,»-* 9JL 



v>(£,t)dt J f[ 9i 

Tl k , m 



t 



dio. 



Setting x = g^ 



t 



we have 



i'x = (g&Ygz 



t 



= e 



t 



= t. 



Hence, by the Fubini theorem, (4.29) reads 

1 



&n,n— k 



d t / <p(€,t'x)f(%)dx= / (p(x)f(x)dx. 



Vn,n-k fOt,, 



an„ 



D 



Lemma 4.13. The dual Radon transform commutes with the group M(n,m) of 
matrix motions. Specifically, if gx — jx{3 + b where 7 G 0(n), fi G 0(m), and 
b G fffln,m> then 

(4.30) ^ogY{x) = ^{gx). 
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More precisely, i/r = r(^,i), then 

(<pog)(t,t) = <P(rt,tP + ?'y'b) 

and 

(4.31) ( i po g y(x)^ip( 1 xl3 + b). 

Proof. By (4.27), 

(^°5) V W = — ^— i <p(rt,?x0 + S'T'b)d£ (7*-0 

&n,ri—k J 

1 

0~n,n—k 

Vr, 



<p{t,g(ixl3 + b))d£ = $frxl3 + b). 



D 



Corollary 4.14. 1/7 g 0(n) and ^(7$, t) = y(£,t), ifrerc <£(7#) = ^(x). 
4.5. Radon transforms of radial functions 

In this section, we show that the Radon transform and the dual Radon trans- 
form of radial functions are represented by Garding-Gindikin fractional integrals 
studied in Chapter 2. This phenomenon is well known in the rank-one case when 
diverse Radon transforms of radial functions are represented by the usual Ricmann- 
Liouville fractional integrals. In the higher rank case, an exceptional role of the 
Garding-Gindikin fractional integrals in the theory of the Radon transform on 
Grassmann manifolds was demonstrated in [GR], We recall (see Section 1.7) that 
a function f(x) on 9Jl n ,m is radial if it is O(n) left-invariant. Each such function 
has the form f(x) = fo(x'x). In the similar way one can define radial functions of 
matrix planes. 

Definition 4.15. For < k < n and m > 1, let T be the manifold of matrix 
planes r = r(£,t), (£,i) g V n , n -k x 9Jt„_ fe , m ; see (4.1). A function ip(r) = <p(£,t) is 
called radial if it is 0(n) left-invariant in the £ variable, i.e., y(7£, i) — <p(£,t) for 
all 7 g 0(n) and all (or almost all) £ and t. 

Note that if tp is a radial function, then 
(4.32) <pin&, W) = <P{& *)> v 7 e 0(n), 6 g 0(n - k). 

This equality is a result of parameterization which is not one-to-one; see Section 
4.1. 

Lemma 4.16. Every function ip on % of the form ip(£,t) = ipo{t't) is radial. 
Conversely, if 1 < k < n — m and <p(£,t) is radial, then there is a function <fo(s) 
on V m such that </?(£,£) = (po(t't) for all £ g V n ; a -k and all matrices t g 9Jt„_fc ;TO 
(if if is a "rough" function then "all" should be replaced by "almost all"). 

Proof. The first statement is clear. By the polar decomposition (see Appendix 
C, 11), for all matrices t G DJl n -k. m we have 

t = US 1/2 , UGV n -k,m, S = t't eV m - 

Let 

t 1 1 ., 

?0 = j £ VVn-fc, MO — j g V n -k,m- 
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We choose 7 e 0(n) and 9 G 0(n — k) so that u = 8'uq, j£6' = £o- Then 

<p(£,t) = ¥>(£,0W /2 ) 

(4-32) . ca , i/ 2 x 

= <f(£o,u s 1/2 ) 

dcf , . 

^o(s). 
This is what we need. 



□ 



Theorem 4.17. Let f be a radial function on 9Jt„ ]m so that f(x) = fo{r), 
r = x'x. Let I J /o be the Carding- Gindikin fractional integral (2.34). Then 

(4.33) f%t) = 7T km / 2 (L k _ /2 f )(s), S = ft G V m , 

provided that either side of this equality exists in the Lebesgue sense. 
PROOF. The statement follows immediately from (4.10): 



(4-34) f(£,t) 



Hi 



U) 

t 



du = 



fo(u'uj + t't)doj 



M k , m 

^ /3 (^/o)(»). 



rot* 



D 



Remark 4.18. The rank of s in (4.33) does not exceed m. If rank(s) = m then 
s G V m . If rank(s) < m (it always happens if k > n — m) then s is a boundary 
point of the cone V m . The fact, that for radial / the Radon transform / is also 
radial, follows immediately by Corollary 4.6, Definition 1.14, and Definition 4.15. 

Let us pass to the dual Radon transform. 

Theorem 4.19. For (£,£) GV n ,n-k x tOtn-k,m, let tp(£,t) = tp (t't). Suppose 
that 1 < k < n — m and denote 



fco(*) = |*|V>(*)> 6 = (n-k)/2-d, d = (ro + l)/2, c 



km/2 
n u n — k.rn 



Let I, $0 be the Carding- Gindikin fractional integral (2.32) of $0. Then for any 
x G yR n ,m of rank m, 

(4.35) ^(x) = c|r| d -"/ 2 (4 /2 $ )W, r = x'x G P m , 

provided that either side of this equality exists in the Lebesgue sense. 
Proof. By (4.27), 



<t>(x) = / <Po(%' 'j£,oCl ' x ) d l ; £0 

SO(n) 




J-n—k 



^ vn,n—k' 
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We write x in the polar coordinates x = vr 1 ' 2 , v E V n , m , r G V m , and get 



(4.36) 



SO(n) 
1 



0"r, 



Mr 1/2 v'toCvr 1/2 )dv. 



Ki, 



Since n — k > m, one can transform (4.36) by making use of the bi-Stiefel decom- 
position from Lemma 1.13. Let us consider the cases k < m and k > m separately. 
1°. The case k < m. We set 



i(I m - a'a) 1 ' 2 



a e 9JT fc , m , u eV r 



n—k,m- 



Multiplying matrices and using (1.40), we obtain 

(p(x) = C\ I \I m - a'a\ 5 ip (r l ^ 2 (I m - a'a)r 1/2 )da 

0<a'a<I m 

= ci J \I m -bb'fMr 1/2 (Im-bb'y/ 2 )db, 

0<bb'<I m 

where c\ — <J n -k,m/<^n,m ■ Then we pass to polar coordinates 

b = vq 1/2 , veV m .k, qeVk, 
and use the equality \I m — bb'\ = \Ik — b'b\. This gives 

ik 
${x) = 2- fe Cl J dv[\q\( m - k - 1 V 2 \I k -q\ 6 ipo(r 1/2 (Im-vqv')r 1 ' 2 )dq 
v m , k 

Ik 

= 2- k Cl \r\- S f dv f\q\^ m - k - 1 ^ 2 ^ (r 1/2 (I m -vqv')r 1/2 )dq. 

V m , k 

Hence, by (2.15), 

0(3) = 7r fem / 2 Ijlzh^ |r| d -"/ 2 (4 /2 $ )(r), 

and (4.35) follows. 

2°. The case k > m. Let us transform (4.36) by the formula (1.41). We have 



${x) = c 2 / \s\-«\I m - S \ 5 Mr 1/2 (I m -s)r 1/2 )d S 



o 

I,n 

= c 2 J\I m -.s\'>\ S \ S Mr 1/2 sr 1 / 2 )d S , 



9 — ra — _ 

^ u k,ni Vn — k.m j /<-» i 

C 2 = ■ — , 7 = k/2 - a. 
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By changing variable r^-^sr 1 / 2 — so (see Lemma 1.1 (ii)) so that \s\ = \r\ 1 \so\, 
dso = \r\ ds, and 

\I m -s\ = |r-V2( r - r l/2 ar l/2 )r -l/2| = | r |-l| r _ Sol 
we obtain 

r 

<f(x) = c 2 |rp 7 ~ <5 ~ d / Ir-soPM* </?o(so)ds 



r 

= c 2 \r\ d - n ' 2 [\r-s \ k / 2 - d $ (so)ds . 
o 

By (1.37) this coincides with (4.35). D 

Remark 4.20. In Theorem 4.19, we have assumed 1 < k < n — m, although, 
Definition 4.10 is also meaningful for k > n — m. In the last case the integral on 
the right-hand side of (4.35) requires regularization. Since the main object of our 
study here is the Radon transform /(£,£), and the condition k < n — m constitutes 
a natural framework of the inversion problem (see Theorem 4.9), we do not focus 
on the case k > n — m and leave it for subsequent publications. 

4.6. Examples 

Formulas (4.33) and (4.35) allow us to compute the Radon transform and the 
dual Radon transform of some elementary functions. The following examples are 
useful in different occurrences. As above, we suppose 

/ = f(x), X e ffln.m, <P = ¥>(£) *), (£i *) € V ntn -k X 9^n-fe,m, 

and write / — ► ip (ip — > f) to indicate that ip is the Radon transform of / (/ is the 
dual Radon transform of ip). Given a symmetric m x m matrix s and a G C, we 
denote 



(4.37) 




s E V m , 



Lemma 4.21. Let 



The following formulas hold. 

(4.39) \x'x\- x / 2 £> Ai |t't|( fe - A )/ 2 , 

(4.40) \I m + x'x\- x ' 2 ± X, \I m + i't^ 2 , 

(4.41) (a - x'x)^- k)/2 - d -^ Ai (a - ftf' 2 " 1 , aeV m 
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Proof. We denote 

h{x) = \x'x\~ x / 2 , 

/*(*) = (a-x'x)i x - k ^- d . 
Then /;(£,*) = ^(i'i) where by (4.34), 

tpi(s) = / \u>'u> + s\~ x/2 duj, 
M k , m 

<p 2 (s) = / \I m +uj'uj + s\- x/2 duj, 
M k , m 

</? 3 (s) = / (a — oo'u) — s) + duj. 

M k , m 

Owing to (A.6), y>i(s) = Ai|s|( fe_A ^ 2 and <^ 2 (s) = Ai|I TO + s|( fe_A ^ 2 . For the third 
integral we have ips (s) = if a — s $■ V m - Otherwise, 

ip 3 {s)= f \a^Lu'LU-s\^- k ^ 2 - d dLU, 

and the formula (A. 7) gives ips(s) — \i\a — s\ x / 2 ~ d . D 

Lemma 4.22. Let 1 < k < n - m, 

(aao\ x r m (rc/2)r m ((A-fc)/2) 

(4.42) A 2 = / Wo ^ 77 77-7777, Re\> k + m-1. 

T m (A/2)r m ((n - k)/2) 

Then 

(4.43) \t't\ {x - n ^ 2 ^ \ 2 \x'x\^- n ^ 2 , 

(4.44) \t'4 x - n ^ 2 \I m + t't\- x ' 2 ^ \ 2 \x'x\( x - n ^ 2 \I m + x'x^-^' 2 , 

(4.45) \t'4 k -^l 2+d {t't - a) (X - k),2 - d X A 2 |.T'a;| d -"/ 2 ( a; ' a ; - a)+ /2_d . 
Proof. Let 

¥>!&*) - |i't|( A -")/ 2 , 

<^(£,t) - |i't|( A - n )/ 2 |/ m + t't|- A / 2 , 

By (4.35), 

(4.46) (p i {x)=c\r\ d - n l 2 {I k J 2 $ i ){r), r = x'x, i= 1,2,3, 



4.6. EXAMPLES 73 

where c = 7r fcm / 2 cr„_fc, m /o-„, m , 

$!(«) = | s |(A-fe)/2-d ; 

$ 2 («) = | S |( A - fe )/ 2 - d |/ m + S |- A / 2 , 

*aW = (s a)i A - fe)/2 - d . 
For ipi(x), owing to (4.46) and (2.32), we obtain 

<p x {x) = c\r\ d - n !\- km ' 2 J \r-(Jw\^- k ^' 2 - d du. 

Hence (A. 7) and (1.37) yield 

r m (A/2) T m ((n- fc)/2) 

This coincides with (4.43). 

For tf>3(x), according to (4.46), we have 

&,(&) = c|r| d -"/ 2 7T- fem / 2 /" (r - w'w - a)^/ 2 -^. 

Hence (p% (x) = if r — a ^ 7-*™, and 

^ 3 (a;) = c | r |rf-™/2 7r -fem/2 /* |r-a-w'w|( A - fe )/ 2 - d rfw 

{o;GU7tfc |T n: uj'uj<r— a} 

if r — a £ Pm. Applying (A. 7), we obtain 

fr(*) = r m iwolr m !! A "5/o! M*-"" k - «l A/2 - d . 

r m (A/2) T m ((n- fc)/2) 

This gives (4.45). 

In order to prove (4.44), we consider the cases k > m and k < m separately. 
1°. The case k > m. By (4.46), 

r 

fr( x ) = Cl |r| d -"/ 2 /| S |( A - fe )/ 2 - d |r- S | fe / 2 - d |/ m + S r A / 2 ds 



o 

I m +r 



= cM d - n/2 |r + / ro - 5 | fe / 2 - d | S -J ro |( A - fe )/ 2 - d | S |- A / 2 d S , 

i,„ 
ci = 2~ m a n -k,m<Jk,ml ' o n ,m- Owing to (A. 3), we obtain 

<h(r) = ciB m (k/2, (A - k)/2) |rf A -"»/ 2 \I m + r |< fc - A >/ 2 , 

and (4.44) follows. 

2°. The case fc < m. By (4.46) and (2.19), 

£2(3) = c |r| d -"/ 2 (4 /2 $ 2 )(r) = c |r|( fc -")/ 2 (/! /2 G 2 )(r- 1 ) 

where 

G 2 ( s ) = lal"*/ 2 -^^- 1 ) = |J ro + s |- A / 2 . 
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Hence, in view of (2.19), 

£ 2 (x)=C7r- fem / 2 |rf fe -")/ 2 I \I m + r- 1 +uj l uj\- x ' 2 du J . 

m k , m 

This integral can be evaluated by the formula (A. 6), and we obtain 

M*) = CT i%2) 2) |r|(fc "" )/2 \^+r-T- X)/2 
= A 2 |r|( A -")/ 2 |/ m +r|( fe - A )/ 2 , r = x'x. 



4.7. Integral identities. Existence of the Radon transform 



□ 



4.7.1. Integral identities. It is important to have precise information about 
the behavior of the Radon transform at infinity and near certain manifolds. The 
same is desirable for the dual Radon transform. Examples in Section 4.6 combined 
with duality (4.28) give rise to integral identities which provide this information in 
integral terms. For m = 1, similar results were obtained in [Ru4]; see also [Ru5] 
for Radon transforms on affine Grassmann manifolds. 

We present these results in two theorems. The constants Ai and A 2 below are 
defined by (4.38) and (4.42), respectively, d = (m + l)/2, and all equalities hold 
provided that either side exists in the Lebesgue sense. 

Theorem 4.23. If 1 < k < n - m, Re A > k + m, - 1, then 

1 



d£ / f(Z,t)\t't\ {X - n)/2 dt 



(4.47) 



v»,„-» OT„_ fc>r , 



= A 2 / f{x)\x'x\ {x - n)/2 dx, 

Wl n , m 

' ' <% [ f(^t)\t't\^- n ^ 2 \I m + t't\-^ 2 dt 



(4.48) 



Vn,n-k 9Jl„_ fc , r , 



A 2 / f(x)\x'x\^- n ^ 2 \I m + x , x\^ k - x y 2 dx, 
' ' ' # f f{i,t)\t't\- s {t't-a) {X - k)/2 - d dt 



@n,n—k 



(4.49) 



A 2 / f(x) \x'x\ d ~ n ' 2 (x'x - a)+ /2_d dx, 
Wl n , m 

S = (n - fc)/2 - d. 
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Theorem 4.24. If 1 <k <n-m, ReX> k + m-1, then 
<p(x) \x'x\~ x/2 dx 



Wl„ 



(4.50) 



0"n, n—k J J 

Vn.n-k M^ k , m 



(4.51) 



fffln.m 

Al 



cp(x) \I m + x'x\ x/2 dx 

d{ f ^(e,t)|/ m +i't| ((fe - A)/2 rft, 



&n ,n — k 

v„,„-k dJl n _ k ,„ 



ip(x) (a — x'x)\ dx 



(4.52) 



9Jt„,m 

Ai 






d£ / </?(£,£) (a -£'£)+ /2 ~ d dt. 



4.7.2. Existence of the Radon transform. In most of the formulas pre- 
sented above we a priori assumed that the Radon transform is well defined. Now 
we arrive at one of the central questions: for which functions / does the Radon 
transform /(£,£) exist? In other words, which functions are integrable over all (or 
almost all) matrix planes £'x — t in 97l„ ]m ? The crux is how to specify the behavior 
of f(x) at infinity. Below we study this problem if (a) / is a continuous function, 
(b) / is a locally integrable function, and (c) / G L p . 

Theorem 4.25. Let f{x) be a continuous function on 9Jl„. m , satisfying 

(4.53) f(x) = 0(\I m +x>x\- x / 2 ). 

If \ > k + m — 1 then the Radon transform /(£, t) is finite for all (£, t) G V n , n -k x 
9ftn-fc,m- If \ < k + m — 1 then there is a function fx(x) which obeys (4-53) and 

PROOF. It suffices to consider the function f\(x) = \I m + x'x\~ x / 2 . By (4.40), 
/a(£,*) = Ai \I m + t't\ {k ~ x)/2 provided A > k + m-1. This proves the first 
statement of the theorem. Conversely, owing to (4.33), the Radon transform /a(£, t) 
is finite if and only if {I J f\)(t'i) < oo (we utilize the same notation f\ both for 
\I m + x'x\- x / 2 and \I m + r\- x l 2 , r = x'x). If A < k + m - 1, then fj 2 ' f x = °o; see 
Remark 2.20. Thus f\(£,t) = oo for the same A. □ 
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Theorem 4.26. Let f be a locally integrable function on dJl n , m - If 

(4.54) J \y'y\ {k -^l 2 \f(y)\dy<^ for all R G V m , 

{ v edJl n , m :y'y>R} 

then /(£,£) is finite for almost all (£,£) G Vn,n-k x ^n-k,m- If (4-54) fails for 
some R G V m , then /(£,£) may be identically infinite. 

PROOF. STEP 1. Suppose first that / is a radial function so that f(x) = /o(r), 
r = x'x. Then (4.54) becomes 

do 

(4.55) ( \r\ k ' 2 - d \f {r)\dr<oQ for all i? G 7> m . 

R 

By Theorem 4.17, 

f(U)^^ km/2 (I- /2 .fo)(t't), 

and therefore, /(£,£) < oo if and only if (/_' fo)(t't) < oo. Now the result follows 
from Lemma 2.19 and Remark 2.20. 

STEP 2. The general case reduces to the radial one. Indeed, let 

F(x)= J /(7*)d7 

SO(n) 

be the "radialization" of/. If (4.55) holds, then, by STEP 1, the Radon transform 
of F is a radial function of the form F(£,t) — &o(s), s = t't, which is finite for 
almost all t. On the other hand, owing to commutation (4.18), &o(t't) is the mean 
value of /(£, t) over all £ G I4,n-fc, and therefore, 

' ; fa,t)di = ^{t't)< 



v n ,„_ fc 
for almost all t. It follows that /(£, i) < oo for almost all (£, t) G V n ,n-k x 9Jtn-fc.m- 

'n 

Theorem 4.27. Let / G L v (dTi, hm ). The Radon transform /(£,£) is finite for 
almost all (£, t) G V n .n-k x %ftn-k,m if o,nd only if 

(4.56) l<P<Po- "^ m ~ 1 1 . 

fc + m — l 

PROOF. By Holder's inequality, 

/ \y'y\ {k - n)/2 \f(y)\dy<A\\f\\ p , 

{yeTl n , m :y'y>R} 
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where 



A p = J \y'y\ p '^ n)/2 dy 



{yeffl n , m :y>y>R} 

DC 

c f | r |p'(fc-n)/2+n/2-d dr ^ = ( m + ^^ c = ^ m )) 



R 
R- 1 

= c f \r\P'^- k ^ 2 - n / 2 - d dr<oo 

o 

provided p < po- Thus, by Theorem 4.26, the Radon transform /(£, t) is finite for 
almost all (£,t) G V n . n -k X 9)tn-fc,m- Conversely, if p > po, then one can choose A 
satisfying 

n + m — 1 

<A<fc + m — 1. 

P 

For such A, the function f\{x) = \I m + x'x\~ x / 2 belongs to L p {Wl rh , m ) (use the 

formula (A. 6)), and f\(£,, t) = oo; see the proof of Theorem 4.25. In order to cover 

the case p = po, we need a more subtle counter-example. One can show that for 

V > POj the function 

(4.57) F(x) = |2/ m + x'x\^ n+m -^ 2p (log \2I m + x'x^ 1 

belongs to L p (VJl n , m ), and F(£,t) = oo. The proof of this statement is technical, 
and presented in Appendix B. □ 

Remark 4.28. (i) Theorems 4.25 and 4.27 agree with each other in the sense 
that all functions satisfying (4.53) belong to L p (dJl n , m ) where 

n + m — 1 

t <P<Po- 

The last statement holds by the formula (A. 6), according to which the function 
\I m + x'x\~ Xp / 2 is integrable for such p. 

(ii) The equality (4.48) implies part (ii) of Theorem 0.5 and provides alternative 
proof of the "if part" of Theorem 4.27. Indeed, choosing A > max{n, k + m — 1}, 
we have \x'x\^ x - n ^ 2 < \I m + x'x^-^/ 2 , and 

— ^— I « / i/«,t)ii* , t| (A - n)/2 i^+t , *i _V2 * 

&n,n—k J J 

Vn,n-k $H„_ fc , m 

(4.58) 

<A 2 J \f(x)\\I m + x'x\( k - n » 2 dx. 

Wl n , m 

Hence, if \I m + x'x\ a / 2 f(x) G L 1 (9Jt nm ) for some a > k — n, then /(£,£) is finite 
for almost all (£,£) G V n ,n-k x 9ftn-fc,m- Moreover, by Holder's inequality, the 
right-hand side of (4.58) does not exceed A||/|| p , where 

(4.59) A p ' = f \I m +x'x\ p '^ k - n ^ 2 dx. 

m n , m 
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By (A. 6), the last integral is finite when p'(k — n) > k + m — 1, i.e., for p < po. 
Thus the left-hand side of (4.58) is finite too, and therefore, the Radon transform 
/(£,£) is well defined for almost all (£,t) G V n ,n-k x 9Rn-k.m- 

4.7.3. Existence of the dual Radon transform. The special case A = 
k + 2d in (4.52) deserves particular mentioning. In this case 

(4.60) / ip{x)dx = Al / di f ip{i,t){a-t't) k ' 2 dt, 



Ai \a 



fe/2 



0~n.n—k 
X ' X<a VVn-fc $H„_ fe , m 

and therefore, 

(4.61) J \$(x)\dx<c I # I \<p(£,t)\dt, 

J J J On,n—k 

x'x<a V nn -k t't<a 

Since a G V m is arbitrary, this implies the following. 

Theorem 4.29. If tp(£, t) is a locally integrable function on the set V n . n -k x 
9Kn-fe.m; 1 < k < n — m, then the dual Radon transform <p(x) is finite for almost 
all x G 9Jl n m ■ 



CHAPTER 5 



Analytic families associated to the Radon 

transform 

5.1. Matrix distances and shifted Radon transforms 

In this section, we introduce important mean value operators which can be 
called the shifted Radon transforms. Here we adopt the terminology of F. Rouvicrc 
[Rou] that admirably suits our case. In contrast to the rank-one case to = 1, where 
averaging parameters are positive numbers, in the higher rank case to > 1 these 
parameters are matrix- valued. 

Some geometric preliminaries are in order. We shall need a natural analog of 
the euclidean distance for the space 9Jt n . m of rectangular n x to matrices. Unlike 
m = 1, when the distance is represented by a positive number, in the higher rank 
case our "distance" will be matrix- valued and represented by a positive semi-definite 
to x to matrix. 

Definition 5.1. A matrix distance between two points x and y in 9H„. m is 
defined by 

(5.1) d(x,y) = [(x- y y(x-y)} 1 / 2 . 

Given a point x G 9H„ im and a matrix fc-planc r = t(£,£) G % (see (4.1)), a matrix 
distance between x and t is defined by 

(5.2) d(x,r) = [^x-ty^x-t)}^ 2 . 
Abusing notation, we write (cf. (3.1)) 

(5.3) \x-y\m = det(d(x,y)), \x - r| m = det(d(x,r)). 

Let us comment on this definition. We first note that for to = 1, (5.1) is the 
usual euclidean distance between points in M n . If y = 0, then d(x, 0) = {x'x) 1 / 2 . 
This agrees with the polar decomposition x = vr 1 ' 2 , r — x'x, v G V n . m . If 
rank(x) = m, then d(x, 0) = r 1 ' 2 is a positive definite matrix. If rank(x) < to — 1 
then d{x, 0) G dV m and det(d(x, 0)) = 0. 

Let us explain (5.2). For £ G V n ,n-k, let {£} denote the (n — fc)-dimcnsional 
subspace spanned by £, and Pr{^} the orthogonal projection onto {£} which is a 
linear map with n x n matrix ££'. Then, as in the euclidean case, it is natural to 
define the distance between the point x G $H„ iTO and the plane r = t(£, i) G T as 
that between two points, namely, Pr^ja; = ££'a; and ££. By (5.1), we have 

d(x,r) = d(^,£i) - [(#* - #)'(#* - m 1 ' 2 = i^'x - tygx - 1)] 1 ' 2 . 

This can be regarded as a matrix distance between x and the projection Pr T a; of x 
onto the plane r. 

79 



80 5. analytic families associated to the radon transform 

Lemma 5.2. 
(i) The group M(n, m) of matrix motions, 

Wln, m 3x — >jx/3 + b, 7 g 0(n), (3 g 0(m), b e Mn, m , 

acts on i/ie manifolds 9Jl n , m and T transitively. 

(ii) TTie determinants \x — y| m and |x — r| m are invariant under the action of 

M(n,m). Namely, 

\gx - gy\ m = \x - y\ m , \gx - gr\ m = \x - r| m , g e M(n,m). 

(iii) TAe distances d(x,y) and d(x,r) are invariant under the subgroup M(n) of 
M(n, m), acting by the rule x — ► jx + b, 7 G 0(n), b G %R n , m (*- e - fl = 0)- 

Proof. The statements follow immediately from Definition 5.1, by taking into 
account that 

(5.4) g: r (£,t)— T (7£,t/? + £V&). 

D 

Definition 5.3. Let r G V m - The shifted Radon transform of a function f(x) 
on 9JT„ m is defined by 



1 / 


dv f (gt 

fCA y- -I- vr 1 / 2 )n 


LU 

t + vr 1/2 


&n—k,m J 

v„_*,„ 

1 / 


Cn—k.m J 

V n -k, n 


J \<i: L 1 VI ) au 7 



(5.5) f r (tt) = / dv / f[ 9i +J _ i /2 do; 



(5.6) 



where (£,t) G V n .n-k x 9Kn-fc,m, 55 G SO(n) is a rotation satisfying (4.9). 

If r = then f r (£,t) coincides with the Radon transform /(£,£); cf. (4.10). 
The integral (5.5) averages f(x) over all x satisfying £'x = t + vr 1 ' 2 . This means 
that / r (£, t) is the integral of f(x) over all x at matrix distance r 1 ' 2 from the plane 
t = t(£,£) (see Definition 4.1). One can write 



(5-7) /r(r) = / r (e,*)= y /(*)■ 

rf(a:,r)— r 1 / 2 

Definition 5.4. Let I < k <n - m, re V m , % be the manifold of all matrix 
fc-plancs 

t = r(£, t) = {x G 9Jt„, m : £'x = £}, (£. f ) G K,n-fc x 9^n-fe,m- 

Given a point z G 9Jl n -k,m at distance r ' 2 from the origin (i.e. z'z = r), the shifted 
dual Radon transform of a function <£>(t) = </?(£, t) on X is defined by 



(5.8) (p r {x) = / tp(£,?x + z)d$. 

v„,„_ fc 
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Let us comment on this definition. We first note that (5.8) is independent of 
the choice of z with z' z = r. Indeed, by passing to polar coordinates 





z = 6u r 1/2 7 6eO{n-k), 



u 



l n 



eV, 



n—k,m ) 



we have 



(p r (x) = / (fii^Z'x + duor 1 / 2 )^ (set $ = rfl>) 

&n,n—k J 

V n , n - k 

<p(r]6',d(ri'x + u r^ 2 ) dr] (use ip(£6', 6t) = tp(£, t)) 



®n,n—k 

Vn,n-k 

(5.9) = / Lp{rj,rj' x + UQr 1 ' 2 )^. 

0~n.ri—k J 

V n ,n-k 

If r = then (p r (x) is the usual dual Radon transform (4.27). Owing to (5.2), 
the matrix distance d(x,r) between x and r = t(£,£'x + z) is r 1 ' 2 . Hence, <p r (x) 
may be regarded as the mean value of <p(£, i) over all matrix planes at distance r 1 ' 2 
from x: 

(5.10) <f> r (x) = J y{r) . 

d(x,r)— r 1 / 2 

5.2. Intertwining operators 

Given a sufficiently good function w(r) on M + , we define the following inter- 
twining operator 

(5.11) (Wf)(T) = (Wf)(tt) = J f(x)w(\?x-t\ m )dx 

0Jl n , m 

f(x)w(\x -T\ m )dx, 

which assigns to a sufficiently good function f(x) on VJl n . m a function (Wf)(r) on 
the manifold X of matrix fc-planes r — r(£, t), (£, t) £ V n ,n-k x 9Rn~k,m- The dual 
operator, which maps a function <p(r) on X to the corresponding function (W*y>)(a;) 
on 9Jl„ ;TO , is defined by 

(5.12) (W*<p)(x) = —^— f dC f <p{Z,t)w{\?x-t\ m )dt 

ip(r)w(\x - r\ m )dr. 



0~n,n—k 

X 

We recall that \x — r\ m denotes for the determinant of the matrix distance between 
x and r; see (5.3). For m = 1, operators (5.11) and (5.12) were introduced in 
[Ru4]. The following statement follows immediately from Lemma 5.2. 

Lemma 5.5. Operators W and W* commute with the group M{n,m) of matrix 
motions. 
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Lemma 5.6. If t = r(£,i), (£,t) e K,n-k x 97l„_ fc , m7 tfien 

(5.13) (H7)(r) = (H7) (£,*)= / /U*)m*-«lrr.)(fa, 

9Jt n _*,„, 
provided that either side of (5.13) is finite for f replaced by \f\. 
Proof. Let g^ G SO(n) be a rotation satisfying 





In-k 



G V„ ,„_ 



.9?£o = £, £o = 

The change of variable x = g^y in (5.11) gives 

(W7)(£,t)= / /(»€») w(|^o»-*U)d|/, Co 

m n , m 

By setting 



we have 



y 



(Wf)(U) 



z 



LU £Wl 



k , m • 



z e Wl n _ 





*-n—k 



k.rnt 



eV n . 



n—k- 



w(\t - z\ m ) dz I f [ g(. 



2; 



do; 



9Kn- 



/(£, z)w(|i - z| m ) dz. 



Lemma 5.7. Lef I < k <n- m, S — (n — k)/2 — d, d = (m + l)/2, 



D 



r = r(£, £), (£, t) G V ni „_ fe x 9Jt„„ fc ,m, "0 





1 m 



gK- 



n—k.m- 



Then 

(5.14) (W7)(£,*) = 2~ m a n - k , m J \r\ s w(\r\V 2 )f r (£,t)dr, 

Vm 



(5.15) (^V)(x) - 2-" l a„_ fc , m / |rf «>(M 1/2 )£r(aOdr 



PROOF. From (5.13), by passing to the polar coordinates (see Lemma 1.11), 
we obtain 



(Wf)(U) 



/(£,* + z)w(\z\ m )dz 



Wl„ 



2~ m / \r\ s w{\r\ 1/2 )dr \ f(£,t + vrV 2 )dv. 
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By (5.6), this coincides with (5.14). Furthermore, (5.12) yields 
(W*ip)(x) = —^— f di [ <p(S,t + S'x)w(\t\ m )dt 

Vn.n-k Wl n - k , m 

d£ ( \r\ s w(|rf/ 2 ) dr f tp{^ur 1 ' 2 + £x) du 



V„,„_ fc V m V„_ 



2 '""'"-'■■•'" ' <% I \ r \ s w(\r\^ 2 )dr I <p(£, Ouor 1 / 2 + ?x) d9. 



&n ,n — k 

V„,„- fc V m O(n-k) 

Now we change the order of integration and replace £ by £9'. Since <p(£9',9i) = 
<p(£, t), then 

(W*cp)(x) = 2 ~ m<Jn - k > m f \r\ 5 w(\r\^ 2 )dr f <p(£, u Q r 1 / 2 + ?x) d£. 
v m v„,„_ fc 

By (5.9), this gives (5.15). D 



5.3. The generalized Semyanistyi fractional integrals 

To avoid possible confusion, wc shall discriminate between operators acting on 
yjln.m and the similar operators on 9Jt„_fc. m . As before, the notation I a , A, T>, and 
H. will be used for the Riesz potential, the Cayley-Laplace operator, the Cayley 
operator, and the generalized Hilbert transform on Tl n . m . We write I a , A, T>, and 
H for the similar operators on 9ft n _fc jTO . These will be applied to functions /(£,£) 
and ip(£,t) in the t- variable. We assume 1 < k < n — to, and denote by §(T) the 
space of functions <p(£, t) which are infinitely differentiable in the ^-variable and 
belong to the Schwartz space 9Jl„_fe. m in the i-variable uniformly in £ G V n ,n—k- 

Consider the following operators 

(5.16) P a f = I a f, P a <p = (/» v , 
where / G S(9Jt„, m ), (p G §(T), and 

(5.17) a G C, a ^ n — k — m 4- 1, n — k — m + 2, . . . . 

The right-hand sides of equalities in (5.16) absolutely converge for Re a > to— 1 and 
arc understood in the sense of analytic continuation for other values of a. Below 
we prove a series of lemmas giving explicit representation of operators (5.16) for 
different values of a. 

Lemma 5.8. Let f G S(9tt n , m ), <p e S(X). If 

Re a > to — 1, a j^ n — k — m+ 1, n— k — m + 2, ..., 
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then P a and P a are intertwining operators of the form (5.11) and (5.12), respec- 
tively. Namely, 



(5.18) 
(5.19) 



(P a fm,t) = 



(P<V)(s) 



1 



7n-fc,m(a) 



f(x)\?x-t\%- k - n dx, 



Wl„ 



7n-fc,m(a) 



^4 / VCC, *) le'a? — *| 



ct-\-k—n 



dt, 



Vn,n-k m n . k . 



where d*£ = o- nn _ k d£ is the normalized measure on V n ,n-k and "fn-k,m(<x) * s the 
normalized constant for the Riesz potential on 9Jl n _fe jTO ; cf. (3.4-6). 

Proof. The formula (5.19) follows immediately from the definitions (5.16), 
(4.27), and (3.49). Furthermore, by (5.16), (4.10), and (3.49), we have 



(P a f)(U) 



1 



7r»-k,m(a) 

2JL- 



7n-fc,m(a) 

SOtn, 



|yir*- n di/ / / U 



$H fc 



/(a:) |£'z - i| 



q+A; — n 



t-v 

dx. 



du> 



This proves (5.18). 



n 



Lemma 5.9. Let f e S(9Jtn,m)> <£ G §C^); 1 < fc < n - m. If I is a positive 
integer so that i < n — k — m (cf. (5.17)), then 



(5.20) (P e f)(U) = c e J dz J f(tt-gt 

Wlk+t, m O(n-k) 



h o 

7 



(5.21) (PV)(z) = c t 



Vn, 



d*S J dz J <p[ e, £'z - 7 

OK, m O(n-fe) 



2; 






dj, 



where 
(5.22) 
Moreover, 
(5.23) 



Cl = 2 TT l m ^ )/\- m {— — )■ 



(P°f)(U) = f(U), (PV)(aO = #aO- 



Proof. By (5.16), analytic continuation of P a f and P Q <p reduces to that of 
the Riesz potential on 9Jl n -k, m - One can readily see that /(£,£), defined by (4.10), 
is a Schwartz function in the t-variablc. Thus, 



(5.24) 



*..w 



p*f = i*f, pV = (/V) 



Now (5.21) follows from (3.54), and (5.23) is a consequence of (3.56). Furthermore, 
by (3.54) and (4.10), 
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(p*m,t) = c t j dz j f[u+-r 

m, m O(n-k) 



ch/ 



r, j dz / d~f f \ .g ? 



t + 7 



du>. 



Since 



Hi 



i + 7 



= e* + m 



then (5.20) follows if we change the notation 







OJ 


h 







z 


I u 


T . 







U! 






— » z. 


z 





D 



Remark 5.10. We call P a f and P Q </? the generalized Semyanistyi fractional 
integrals; see Remark 5.14 for comments. Formulas (5.18)— (5.23) can serve as def- 
initions of P a f and P a tp if / and <p are arbitrary locally integrablc functions so 
that the corresponding integrals converge. 

Lemma 5.11. Let / e S(9Jl„, m ), <p e$(1). If t = 1,2,. .., then 

(5.25) (P- 2 V)(e,*) = (-l) m 'AV'(^i), 

and 

(5.26) (p- 2 vxao = (-irnAV] v (*) = (-ir' / A V(e,i) rf*e 

V n ,„_fc 

Proof. The statement follows from (5.16) and (3.60). □ 

Lemma 5.12. Let f £ S(9Jt„, TO ), v? 6 S(T), £=1,2,.... VKe denote 



(5.27) ci 



(_!)"»« T((n- A; -m)/2) 



C2 



(-l) m(£+1) r m ((m + l)/2) 



2»m+l 7r (m+n-fe)/2 ' 

(i) J/ 1 < k < n - m and P ? (£) = A £ /(£, t) then 

(5.28) (p'-^fmt) = i-ir^mt) 



i?/2 



and 
(5.29) 



. : / dv j F((t- vy')dy 

gn-k-l 

m<[rlA/,jV; 



(p'-^ix) = (-ir^A^(x) 



Cl / eU£ J dv I AV(£,t) 



t—^'x — vy' 



dy. 
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(ii) If k = n — m, then 

(5.30) (P 1 - 2 "!)^ t) = c 2 (HV 2l - l f{Z, ■))(*) 
and 

(5.31) (P^VX*) = c 2 {HV 2l - l ^-)Y{x) 

= c 2 y (w^-v^,-))(r^)rf4, 

V„,„_ fc 

7i fcmj ifte generalized Hilbert transform; cf (3.50). 

Proof, (i) follows from (5.16) and (3.61); (ii) is a consequence of (5.16) and 
(3.62). □ 

The following statement is the main results of this chapter. 

Theorem 5.13. Let 1 < k < n — m, a e C; a ^ n — k — m+l,n — k — m+2, .... 
(i) Iff£§(S m ) then 

(5.32) (P a f)(x) = C n , k , m (I a+k f)(x) 
(the generalized Fuglede formula), where 

(5.33) c nXm = 2 km ir km / 2 T m (|) /T m (^^j . 

(ii) Let Iq = min{?7i, — l,n — fc — m}, 

21 = {0, 1,2, . ..,£0} U{a : Rea>m-1; a^n- k-m + l,n- k— m+2, . .. }. 

Suppose that f G Lj oc (VJl n . m ) and a € 21. TTien equality (5.32) holds provided 
that the Riesz potential (I a+ f)(x) is finite for f replaced by \f\ {e.g., for f G L p , 
1 < p < n/ (Re a + k + m — 1)). 

PROOF, (i) Let / e §(S m ). We make use of the equality (4.47) with A = a + k 
and Re a > m — 1 . This gives 

—*— I di f f(U)\tC +k - n dt 



(5.34) 

f(y)\y\? n +k - n dy. 



T m (n/2)T m (a/2) , ,. ,„+,,_„ 



r m ((a + k)/2)T m ((n-k)/2) 

Replacing f(y) by the shifted function f x {y) = f(x + y) and taking into account 
(4.19), we get 

—^— [ <% [ m^'x + t)\t\^+ k - n dt 

Vn,n-k Wl , 

' *'*' *■ n — k . m 



(5.35) 

f(x + y)\y\^ +k - n dy, 



r m (n/2)T m (a/2) , n ,,-/,-, 



r m ((a + fc)/2)r m ((n-fc)/2) 
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cf. (5.19) and (3.49). Hence, (5.32) follows when Re a > m — 1 with the constant 
_ T m (n/2) r m (a/2) y n . m (a + k) 



Cn.k.r 



r m ((a + k)/2) T m ((n - k)/2) 7 „_ fc , m (a) 



( 3 - 46 ) okmkm/2-r ( n \ i r 

= 2 it < L m I — I /!„ 



n — k 



By analytic continuation, it is true for all a G C, a. ^ n — k — m+1, n — k — m+2, .... 
(ii) Suppose / G Ij^SDtn,™). For Re a > m - 1, (5.32) follows from (5.35) by 
taking into account that (5.35) was derived from (4.47), and the latter is also true 
for locally intcgrablc functions. For a = £, £ = 1, 2, . . .£q, we have 



(pV)(*) 



Q 



&n,n—k 



d£ dz d"f f \ x ~ 9£ 



Vn,n-k m im O(n-fc) 9JL 



w 




z 




7 








da; 



q / dp I dz I d-y 

G(n) Wl im O(n-k) (ffl 

where q is the constant (5.22). We write 



/ U - p 



LO 




z 




7 








LO 






o> 






ly 


n 








z 




= 








z 


7 









(J 


7 _ 








du>, 



Then the change of variables /3 



4 o 

7 



/3 gives 



(P7)(a0 = Q j dz y dLo J f\x-p 



z 




d/3 



Q 



dy / / [x-P 



M e+k . m o(, 



d/3 



where 



Cn,k,m(I + f)(x) 



C-n.k,', 



Cl 2 (l+k)m ^ +k)m/2 T m r-)/T ri 



-£-k 



fon fcm/2-p ( n \ m ( U * 

= 2 tt /r m y/r m i^- 



D 
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Remark 5.14. Some comments are in order. The idea to study Radon trans- 
forms as members of the corresponding analytic families goes back to Semyanistyi 
[Se] who considered the hyperplane Radon transform on R™ (the case m = 0, k — 
n — 1). This approach was extended by Rubin to Radon transforms of different 
kinds, see, e.g., [Ru2]. In the rank-one case m — 1 the formula (5.32) is due to 
Fuglede [Fu] for a = and to Rubin [Ru2], [Ru4] for any a. In the higher rank 
case m > 1, it was established for a = in [OR] (for sufficiently good functions) 
and justified for / G LP in [Ru7]. 

The formula (5.32) has the same nature as the classical decomposition of dis- 
tributions in plane waves. To see this, one should formally set x — in (5.32) and 
regard this equality in the framework of the theory of distributions. The idea of 
decomposition of a function in integral of plane waves amounts to Radon [R] and 
John [Jo], and has proved to be very fruitful in PDE. For the distribution |a;| on 
M. n an account of this theory is presented in [GShl, Section 3.10]. This method 
was generalized by Petrov [P2] for the matrix case for k = n — m, and briefly 
outlined by Shibasov [Sh2] for k < n — m. Our way of thinking differs from that 
in [Sh2], [P2], and [GShl], and the result is more general because we allow / to 
be a "rough" function. 

For the sake of completeness, we also present the following statement which 
follows from Theorem 5.13 and the semigroup property of Riesz potentials. Con- 
cerning this property, see [Kh] and [Ru7] . 

Theorem 5.15. Let 

Re a > m — 1 , Re (3 > m — 1 , Re (a + (3) < n — k — m + 1. 

Lf the integral j a +P+ k j absolutely converges then 

(5.36) P a pPf = c nXm I a+0+k f, 

c n ,k,m being the constant (5.33). 

PROOF. By (5.32) and (5.16), we have 

c n .k, m L a+t3+k f =P a+/ 7 = (L a+t3 fY = {L a PfY =P a pPf. 

D 



CHAPTER 6 



Inversion of the Radon transform 



6.1. The radial case 

Theorems 4.17, 4.27, and the inversion formula (2.51) for the Garding-Gindikin 
fractional integrals imply the following result for the Radon transform of radial 
functions. 

Theorem 6.1. Suppose that f(x) = fo(r), x s 9Jt„ im; r = x'x, and let 

(6.1) /e£ p (an n , m ), l<p< l + m ~\ . 

k + to — 1 

Then the Radon transform, /(£,£) is well defined by (4-10) for almost all (£,£) £ 
Vn, n ~k x OHn-i.m and represents a radial function, namely, 

(6.2) /(£, t) = 7r km / 2 (I k : /2 f )(s) = <po(a), a = ft e V m . 
If 1 < k < n — m, then /o can be recovered from tpo by the formula 

(6.3) h(r)=n- km l 2 (D k J 2 ^){r), 

where D_ is defined in the sense of D' (V m )- distributions by (2.52). 

Remark 6.2. The condition (6.1) can be replaced by the weaker one. Indeed, 
owing to Theorem 2.19, it suffices to assume that 

oc 

(6.4) /|r| (fe - m_1)/2 |/o(r)|dr < oo for all R € V m . 



R 

Furthermore, we know that by Theorem 4.9, the assumption k < n — m is necessary 
for injectivity of the Radon transform. One might expect that once we restrict to 
radial functions, then this assumption can be reduced. However, it is not so, because 
the function ip in the proof of Theorem 4.9 can be chosen to be radial. Note that 
if k > n — to, then the exterior variable s in (6.2) ranges on the boundary of the 
cone, and we "lose the dimension" . 

In the same manner, Theorem 4.19 and Lemma 2.23 allow us to obtain an 
inversion formula for the dual Radon transform. 

Theorem 6.3. Let <p{£,t) = (p {s), (£,£) 6 V„, n _ fe x Wl n - k , m , s = ft. We 
assume 1 < k < n — m and denote 

$o(«) = klVo(s), 5 = (n-k)/2-d, d=(m + l)/2. 

If < &o( s ) € ^locO^m) then the dual Radon transform f>(x) is well defined by (4-27) 
for almost all x G 9Jl n ,m and represents the radial function 

(6.5) <p(x)=c\r\ d - n / 2 (I k + /2 ^ )(r)=fo(r), r = x'x eT m , 
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c = TT krn ' 2 <7 n -k,m/o'n,m- The function ipo can be recovered from /o by the formula 

(6.6) Ms) = c-^iD^-^FoKs), F (r) = \r\ n ^- d f (r), 
for any integer j > fc/2. If k is even, then 

(6.7) Ms)=c- 1 \ S \- 5 (D k + /2 F )(s). 

The differential operator D + in (6.6) and (6.7) is understood in the sense of 
T/'iVm)- distributions (cf. (248) and (249)). 

6.2. The method of mean value operators 

As it was mentioned in Introduction, after the 1917 paper by Radon [R], the 
following two inversion methods have been customarily used for reconstruction of 
functions from their integrals over affine planes. These are the method of mean 
value operators, and the method of Riesz potentials. The third classical method 
presented in [GShl] is the method of plane waves. Below we focus on the first 
method and extend it to functions of matrix argument. 

Definition 6.4. Given a function f{x) on 2Jl„ im , we define 

(6.8) (M r f)(x) = — / f(vr 1 ' 2 + x)dv, r G V m . 

V n , m 

This is a matrix generalization of the usual spherical mean on R". 
We say that r g V m tends to zero when tr(r) — ► 0. 
Lemma 6.5. If f g L p (9Jl nym ), 1 <p < oo, then 

(6.9) lim(M r /)(a;) = f(x) 

i — >o 

in the IP -norm. If f g Co(VJl n ^ m ), i.e., f(x) — ► as \\x\\ = (tr(x'x)) 1 ' 2 — ► cxd, this 
limit is uniform on 9Jl„, m . 

PROOF. By the generalized Minkowski inequality, 

\\M r f - f\\ p < J- / ||/(vr 1/2 + -)-/(-)ll P dt;. 

V„, m 

Since the integrand does not exceed 2||/|| p , by the Lebesgue theorem on dominated 
convergence, one can pass to the limit under the sign of integration. Let y — 
vr 1 ' 2 E Tl n , m - We represent the corresponding nm-vector y — (3/1,1, . . . ,y n ,m) in 
polar coordinates 

y = 9p, p=\\y\\ = (tr(y'y)) 1 / 2 , 6 G S nm '\ 

so that 

Km J/K 1 / 2 + •)-/(•) ll?=lim / \f(0p + x)-f(x)\ p dx = 0. 

This gives (6.9) in the L p -norm. The proof of the second part of the statement is 
the same. □ 
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The following lemma which combines the Radon transform and the shifted dual 
Radon transform (5.7) is the core of the method. It reduces the inversion problem 
for the Radon transform to the case of radial functions. Thus "dimension of the 
problem" becomes essentially smaller. 

Lemma 6.6. For fixed x G ffl n . m , let F x (r) = (M r f)(x), r G V m . Then 

(6.10) (fy s (x)=ir km / 2 (I k _ /2 F x )(s), s€V m , 

provided that either side of this equality exists in the Lebesgue sense. 

Proof. We denote <p(£,t) = /(£,*), f x (y) = f(x + y). Let z G 9Jt n -fc,m be a 
matrix at distance s 1 / 2 from the origin. By (5.8) and (4.19), 

(WW = -^— I !*&*)<% 

V n , n -k 



SO(n) 

Interchanging the order of the Radon transform and integration over SO(n), and 
using (4.18), we get 



tp s (x) = / dw \ f x 7 



U) 

z 



dr/. 



M k , m s °(") 



Hence, z —* <p s (x) is the Radon transform of the radial function 



fx(y)= / fx(iy)d'y= / f(x + "fy)d-f, yeTl 

SO(n) SO(n) 

In other words, 

(6.11) Mx) = (fx) A (£,z), 

where 

fx(y) = — I f(x + vr^ 2 )dv^F x (r), 



0~n,m 

V, 



r = y'y. 



Owing to (6.11), the desired equality (6.10) follows from the representation (4.33) 
for the Radon transform of a radial function. □ 

Corollary 6.7. Let f e L p {m n ^ m ), l<p<(n + m- l)/(fc + m - 1). Then 
(6.10) holds for almost allx G 9Jl„ im . In particular, (6.10) holds for any continuous 
function f satisfying 

f(x) = 0(\I m +x'x\- x / 2 ), \>k + m-l. 

Proof. It suffices to show that the right side of (6.10) is finite for almost all 
x G dJl n ,m whenever / G L p (Wl nym ), 1 < p < (n + m— l)/(k + m — 1). By Theorem 
2.19, the integral (I_ F x )(s) is well defined provided 

oo 

(6.12) /-/MV-'IJW-,!*^ <or,»nev m . 

R 
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The proof of (6.12) is simple. Indeed, taking into account (6.8) and using Lemma 
1.11, we obtain 

oo 

I ■' J— [ \ r \ k / 2 - d dr I \f{vr 1 / 2 +x)\dv 

\y'yf- n ^\f(x + y)\dy. 



R V 7 



{ y eWl n , m --y'y>R} 

By Holder's inequality, / < A||/|| p , where the constant A is the same as in the 
proof of Theorem 4.27. If p < (n + m — l)/(fc + m — 1) then A < oo, and we are 
done. □ 

Now we can prove the main result. 

Theorem 6.8. Let 1 < k < n — m, 

n + m — 1 



(6.13) /eL p ($H„, m ), 1< P < 



k + m — 1 



Then the Radon transform ip(£,t) = f(S,,t) is well defined by (4-10) for almost all 
(£, t) G V-n.n-k x 9Rn-k,m a nd can be inverted by the formula 

( L p ) 

(6.14) f(x) - ir- km / 2 lim(D! /2 $ x )(r), <S> x (s) = <p s (x), 

where D_ is defined in the sense of T)' (V m )- distributions by (2.52). If f is a 
continuous function satisfying 

(6.15) f{x) = 0(\I m +x'x\- x/2 ), \>k + m-l, 
then the limit in (6.14) can be treated in the sup-norm. 

PROOF. By Lemmas 6.6 and 2.24, one can recover F x (r) = (M r f)(x) and get 
(M r f)(x) = ir- km / 2 (D k _ /2 ^ x )(r), r G V m . 

Now the result follows by Lemma 6.5. D 

6.3. The method of Riesz potentials 

The second traditional inversion method for the Radon transform reduces the 
problem to inversion of the Riesz potentials. This method relies on the formula 
(5.32) where a is in our disposal. The case a — corresponds to the Fuglede 
formula 

(6.16) (f) v (x) = c n , k , m (I k f)(x), 
which together with Theorem 3.17 implies the following. 

Theorem 6.9. Let f e L p (97t„ !m ), 1 < p < n/(k + m - 1). Then the Radon 
transform ip = f is well defined, and f can be recovered from ip in the sense of 
$' -distributions by the formula 

(6.17) C „ Am (/» = (^,/- fc 0), 

G $, c n , k , m = 2 km n km / 2 r m (|) /T r , 
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where the operator I~ k is defined by 

{r k ^(x) = (T- 1 \yt l T ( t ) ){x). 
In particular, for k even, 

(6.18) c n , k , m (f, <t>) = (-l) mk/2 &, A fc / 2 0), 

A being the Cayley- Laplace operator (1.44)- 

Remark 6.10. For k odd, the Radon transform can be inverted under more 
restrictive assumptions as follows. Let k < n — to. We choose a = 1 in (5.32) and 
get 

(6.19) I\f = c n ^ m I k+1 f. 

If / € L p (Tl n . m ), 1 < p < n/(k + to), then / can be recovered from <p = / by the 
formula 

(6.20) c nXm (f, 4>) = (-i) m ( fe+1 )/ 2 (/V, & {k+1)/2 <p), <t> e $. 

6.4. Decomposition in plane waves 

This method was developed in [P2] for the case k = n — to, and outlined in 
[Sh2] for 1 < k < n — m. It is based on decomposition of distributions in (matrix) 
plane waves. As we have already noted in Remark 5.14, our formula (5.32) has the 
same nature. Together with (5.26), (5.29), and (5.31), it enables us to invert the 
Radon transform of functions / G §($H„. m ). 

Theorem 6.11. Let 1 < k < n — m, f € §(S m ). The Radon transform 
tp(£,,t) = /(£,£) can be inverted by the following formulas. 
(i) For k even, 



(6.21) f(x) = (-l) rofc/2 c-U / Afe/ M^) 



t=(,'x 



V„ 



c t - 2 km -K km ' 2 V (-\ IV I - - 

(we recall that d*£ stands for the invariant measure on V n ,n-k of total mass 1). 
(ii) For k odd and k < n — to, 



3.22) /(a;) = ci J d.£ J dv I A( fc+1 )/V(C,t) 

V n , n - k S"-*-i 



dy, 

t—^'x—vy' 



Cl = (_ 1 yn(k + l)/2 2 ^k,n- m -l 7T (k-n)/2- m (k/2 + l) rm+i (lLLl\ /T m (- 

(iii) For k odd and k = n — in, 

(6.23) /(aO = C2 | (W©V(£,-))(^K£, 

V n , n -k 



Cl 



{ _ ir{k+ Z)/2 2 - k m n - m{k+m)/ 2 Tm ^ ^ (^+}_^ /Tm g) 
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PROOF. We write (5.32) with a = -k so that 

(6-24) f(*) = c^, m (P- k <p)(x), 

where P~ k is the operator (5.16). Now it remains to apply formulas (5.26), (5.29), 
and (5.31). 

□ 

Formulas (6.21)-(6.23) differ from those in [P2] and [Sh2]. 



APPENDIX A 

Table of integrals 

Let k,m e N; ft,ft)eC; d = (m + l)/2. We recall that S m ,P m , and V m 
denote the set of all m x m real symmetric matrices, the cone of positive defi- 
nite matrices in S m , and the closed cone of positive semi-definite matrices in S m , 
respectively. The following formulas hold. 

b 
(A.l) f\r- a\ a - d \b - rf- d dr = B m (a, 0)\b - a\ a+p - d , 

a 

a € S m , b > a, Rea>d— 1, Rej3>d—1; 

b 

(A-2) [\b- rMr - cf^^- = ^#|6 - a|«+'-', 



\r\ a +^ \a\ a \bf 

C 

aeS m , b>a, c = b 1/2 a 1/2 b- 1 a 1/2 b 1/2 , Rea>d-1, Re/3>d-l; 

b 

(A.3) J \b - r|— |r - 4/^^ = ^pA& - ^1"+^, 

6 > J m , /?e a > d — 1 , i?e /? > d — 1 ; 

DC- 

(A.4) /" |r|-T|r - s\ a - d dr = |s| a -TB ro (a,7 - a), 

s G T^m j R& a > d — 1 , ite (7 — a) > d — 1 ; 

dc 

(A.5) /" \I m + r|-T|r - s| a - d dr = |J m + s| Q -TB m (a,7 - a), 

s G ? m , -Re a > d — 1, i?e (7 — a) > d — 1; 

(A.6) y |6 + ^l-V'dy = 7rW2 ^[i A /2) fcV2) |fc| (fc - A)/2 , 

97l fc , m " 

beP m , ReX>k + m-l; 
95 
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(A.7) J |6 - y'y^^dy = 7rfcm/2 ^^ /2) fc)/2 V | A/2 - d , 

{yeffl k , m : v'v<b} 

beV m , Re\> k + m — 1. 

PROOF 

(A.l). We have 

6 c 

/ s / |r -„M 6 - r r**-/ |s Mc-,r^, 

a 

c = b — a. Let s = c 1 / 2 ^ 1 / 2 . By Lemma 1.1 (ii), 

i m 
I = \c\ a+ti - d J \t\ a ~ d \I m - tf- d dt = \c\ a+fi - d B m {a, (3). 
o 
(A. 2), (A. 3). Let us write (A.l) from the right to the left, and set 

r = aW/V&^V/ 2 , dr = (\a\\b\) d dT. 

This gives 

c" 1 

B m {a,l3)\b-a\ a+fj - d = {\a\\b\) a+fi - d f \t - b' 1 ^^- 1 - T f~ d dT. 

ft- 1 

Then we set r = s _1 , dr = |s| _2d cfs, and get 

6 

a-rf| „i/3-rf ds 



B m (a,p)\b - a\ a+fi - d = \a\ a \bf J \b - s\ a - d \s - c\ 



\a+0 



which was required. The equality (A. 3) follows from (A. 2). 

(A. 4), (A. 5). By setting r — q^ 1 , dr = \q\~ m ~ 1 dq, one can write the left-hand 
side of (A. 4) as 

\ S \ a - d I Itf-*-^- 1 - q\ a - d dq ( V> \s\ a ^B m (a, J - a), 

and we are done. The equality (A. 5) follows from (A. 4) if we replace s and r by 
I m + s and I m + r, respectively. 

(A. 6), (A.7). By changing variable y —* yb 1 / 2 , we obtain 

\b + y'y\- x / 2 dy = \b\^- x ^J 1 , 

\b- 1 /y\^- k ^ 2 - d dy=\b\ x / 2 - d J 2 , 
{yeM k , m - y'y<b} 
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where 

Ji = J \l m + y'y\- x/2 dy, 
wi k , m 

■h = / \I m -y'y\ (x - k)/2 - d dy. 

{yeM k , m : y'y<I m } 

Let us show that 

__ _ 7r km / 2 T m ((X - fc)/2) 
1 " 2 " T m (A/2) 

The case k > m. We write both integrals in the polar coordinates according 
to Lemma 1.11. For J\ we have 

•h = 2- m <Tk,m J ' \r\ k l 2 - d \I m +r\- x ' 2 dr 
.,- -,,, ,-, . k X — k 



'" l 2' 2 
(the second equality holds by (A. 5) with s — 0, a — k/2, 7 = A/2). Similarly, 

In, 

J 2 = 2- m <r fc , m / \r\ k ^- d \I m -r\^- k ^ 2 - d dr 


'& A-fc N 



9 — m d _ 

^ ( - , k,m- D m \ (-. ; (-. 

Now the result follows by (1.13) and (1.37). 

The case k < m. We replace y by y' and pass to the polar coordinates. This 

yields 

Ji = j \l m + yy'\- x/2 dy 
m mik 

= 2- k J dv J \I m +vqv'\- x / 2 \ q \( m - k -VI 2 dq, 

v m , k v k 
(\I m + vqv'\ = \I k + q\) 

= 2- k f y m , k J\ q \^- k -^/ 2 \I k + q \-^d q . 
Pk 
By (A.5) (with s = 0, to = k, 7 = A/2) and (1.10), 

T - 9- k ~ R ( m A_m 



,2' 2 
7r fcm /2 rfc (( A _ m ) /2 ) 

T fe (A/2) 
n km / 2 T m ((\-k)/2) 

r m (A/2) 
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Similarly, 

h = J \i m -yy'\ {x - k)/2 - d dy 

{y£ffl mik : yy'<I m } 

= 2- k f dv f \I m -vqv'\^- k ^ 2 - d \q\^- k -^ 2 dq 

V m ,k {q£V k :vqv'<I m } 

Ik 
= 2- k a m , k f\h g |(A— fc -l)/2 |g | (m - fe -l)/2^ 

'() 



m A — to 

fc-Ofe ' 



2 <7m._k.Bk . 

and we get the same. 



APPENDIX B 

A counterexample to Theorem 4.27 

Let us prove that the function 

(B.l) F(x) = F {x'x) = \2I m + x'x\- (n+m - l ^ 2p {\og \2I m + x'x\y l 

belongs to L p (9Jt„, m ), but 

(B.2) F(£,t) = oo for p>p = (n + m-l)/(k + m-l). 

We shall put " ~ " , " < " , and " > " instead of "=" , "<" , and ">" , respectively, if 
the corresponding relation holds up to a constant multiple. To prove (B.2), owing 
to (4.34), we have 

&(&)= f \2I m +u'uj + S \-^ +m - 1 ^ 2 P{\og\2I m +u'u + S \)- 1 dLo, 

SW».m 

where s = t't. This gives ( set uj = y(2I m + s) 1 / 2 , du = \2I m + s\ k / 2 dy) 
F{£,t) = \2I m + s \ k / 2 -^ n+m - 1 ^ 2 PI 1 (z), z = log\2I m + s\, 

h(z) = J \I m + y'y\-^ +m -^' 2p {z + \og\I m + y'y\)- 1 dy. 

m k , m 

For k > m, by Lemma 1.11, we obtain 

h{z)~ J \r\ k l 2 \I m +r\-( n+m - l V 2 v{z + \og\I m + r\)- l d«r, 

d*r = |r|~(" l+1 )/ 2 dr. Let us pass to polar coordinates on V m - 

(B.3) r — j'a-f, 7 e 0(m), a = diag(«i, . . . ,a m ), a 3 ■> 0. 

Then 

m 

(B.4) d*r = c m v m (a)(j{a^ m+1)/2 da>jd 1 , 

3=1 
where 

m 

v k (a)= [] l«i-°il. k = 2,...,m, c" 1 = tt"^^/ 4 J[j T(j/2), 

l<i<j<k j=l 

[T, p. 23, 43]. By (B.3) and (B.4), 
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oo oo 

h(z) ~ / ... / v m (a) 





z + V^ l°g(l + a>j 



j=i 



[] a ( fc -™-D/ 2 (i + a j )-("+ ro - 1 )/ 2 ^a j 



The transformation a j + 1 = 6j yields 

3 5 oo 

(B.5) A(«) > J b^dbi J bldb 2 . ■ .J v m (b) 

2 4 2m 

where A = (k — m — l)/2 — (n + m — l)/2p. Note that in (B.5), 






m (t0im 



m—1 

v m (b)= n i^-^i= n i^-^i n i^-&,i>(& m -2mr- 1 

l<i<j<m i—1 l<i<j<m—l 

and log&i < . . . < log6 m . Hence, for p > po, 

OO 

h{z) > j & m (6 m - 2m)" 1 - 1 [z + to log ^r 1 rf6 m = oo. 

2m 

If k < to then 

M*)= / l/m+^T^+^^^lz + logl^+^'l)- 1 ^ 

dvf\q\ ( - m - k - 1 ^ 2 \I m + vqv'\- {n+m - 1)/2p (z + log |/ m + vqv'^dq. 

v m , k v k 

Using the equality \I m + vqv'\ = \Ik + q\ [Mu, p. 575], and setting q = (3'bf3, 
/3 e 0(k), b = diag(6i, . . . , bk), bj > 0, we have 



h(z) 



| 9 |(m-fc-l)/2| /fc + q \-(n+m-l)/2 P{z + bg | /fc + ^-1^ 
f TT ^"-fe- 1 )/ 2 ^ + & .-p(n+m-l)/2p 



OO oo 



o o •?= 1 

fe 



3 = 1 



v k (b)dbi . . .db k . 
Proceeding as above, for v = (to — k — l)/2 — (n + to — l)/2p, p > po, we obtain 



h(z) > I ' b v k (b k - 2k) k - x [z + k\ogb k ]- 1 db k = oo. 
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Let I2 = \\F\\^. To complete the proof it remains to show that h < 00. As 
above, we have 

|r|"/ 2 |F (r)|^r 



\r\ n ' 2 \2I m + r|-( n+m - 1 >/ 2 (Iog \2I m + r\)- p d«r 



/m 
[ \{ a «/2-(™+l)/2 (2 + 0i) -(n+ m -l)/2 






u m (a)dai . . . da m , 



where R™ is the set of points a = (ai, . . . , a m ) with positive coordinates. Let us 
split R+ into m + 1 pieces Oq, . . . , fi m , where 



fin 



{a : < Oj < 1 for all j = 1, . . . , m}, 



^m = {a '■ o,j > 1 for all j = 1, . . . , to}, 
and ^£ (^ = 1, 2, . . . , to — 1) is the set of points a G R™ having £ coordinates > 1 

m 

and m — £ coordinates < 1. Then I2 — ^2 At, where Ag = j n (...). 



(The case m = 2) 



fl2 








fii 


r* 2 


1 








O 


Oi 



1 



fll 



For p > 1 , we have 



A) = /.../ [n a ; /2 - (m+1)/2 (2+ flJ r ( " +m " 1)/2 
* =i 

r ~i _ p 

m 

x >_, log(2 + dj) v m (a) da\ . . . da m 

m „ 

< [] / a; /2 - (m+1)/2 da 3 < 00. 
3=1 { 
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In order to estimate A m , we first note that the integrand is a symmetric function 
of oi, • • • ,a m . Hence, A m = [m\)B m , where B m is the integral over the set {a : 
a £ J7 m ; a\ > a 2 > ■ ■ ■ > a m }. Using obvious inequalities 



(B.6) 
we have 



„/2-( m +l)/2 (2 + a . ) _ ( „ +m _ 1)/2 < „ 



B m 5, 



da j 



1 



af log p (2 + ai) 



a\ "rn-2 

\a^ m da 2 ... / a m m ± jj \ai-aj\da m -i 



l<i<j<m— 1 



Since 



a~ m JJ |oj -a m \da r , 



„ rn—1 rn—1 „ 

\ a 7n m Yi l ai - a m\da m < ( Y]_ a, J / 



da r 



i=\ 



i=\ 



1-a 



it follows that 



B m 5, 



dai 



m 



ai Qm-2 

,1— m 



1 — m m— 1 m— 1 

f 1 n«^n< 



i=l 



a? -1 log"(2 + d. 



/ a^ m da 2 ... / a^_^ JJ |oj -aj|da m _i 



l<i<j<m— 1 



Repeating this process, we get 



Bm 5, 



da\ 



ailog p (2 + ai ) 



< oo. 



Let us estimate Ag, £ = 1, . . . ,m— 1. Owing to the symmetry, it suffices to consider 
the integral 



oo ai c^-l 

-B £ = / dai I da 2 - / I J | ^ 

!. I 1 "J = 1 

x J(oi, . . . ,a^) da^, 



oo ai ««— l /j 

fda 1 fda 2 ... J [n a ; /2 - (m+1)/2 (2 + aj r (n+m " 1)/2 ' 



where 



J(ai, . . . ,a*) = /■■■/[ II a; /2 - (m+1)/2 (2 + o.j-^+^-DA 
o o J^+i 



5^1og(2 + Oj-) 



u m (a)da^ + i . ..da m . 
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We use the inequality 

£ m £ 

l<i<j<m i—lj—i+1 l<i<j<£ i—1 

This gives 



J(a u ...,ai)< pog(2 + oi)] p '[I |oi-Oj-|JJ(oj + i; 

l<i<j<£ i=l 

and, owing to (B.6), 

ai 



m-£ 



1 I ! l<i<J<-t:— 1 

^-i £-1 



Proceeding as above (see the argument for i? m ), we obtain 

oo 

a 1 log-(2 1 + a 1 ) <0 °- 
i 

Thus, I2 < oo, and we are done. 



APPENDIX C 



Some facts from algebra 



We recall some well-known facts repeatedly used throughout the monograph. 
We do this for convenience of the reader by taking into account that the literature 
on this subject is rather sparse. More results from matrix algebra can be found, 
i.e., in [Mu] and [FZ]. 

Notation: 

• yftn.m is the space of real matrices x — (xij) having n rows and m columns. 

• x' denotes the transpose of x. 

• rank(x) = rank of x. 

• I m is the identity m x m matrix. 

• tr(a) is the trace of the square matrix a. 

• \a\ is the absolute value of the determinant det(a). 

• 0(m) is the group of orthogonal m x m matrices. 

• Vn t m = {»£ yJin.m ■ v'v = I m } is the Stiefcl manifold. 

Some useful facts: 

1. If x is n x m and y is m x n, then \I n + xy\ = \I m + yx\. 

2. If a is a square matrix and \a\ ^ then (a -1 )' = (a') -1 . 

3. rank(x) = rank(x') = rank(a;a;') = rank(x'x). 

4. rank(xy) < min(rank(x), rank(y)). 

5. rank(x + y) < rank(a;) + rank(y). 

6. rank(ax6) = rank(x) if a and b are nonsingular square matrices. 

7. tr(o) = tr(a'). 

8. tr(a + 6) = tr(a) +tr(6). 

9. If x is n x m and y is m x n then tr(xy) = tr(yx). 

10. Let S m be the space oimxm real symmetric matrices r = (rij), r^j = r^j. 
A matrix r £ S m is called positive definite (positive semi-definite) if u'ru > 
(u'ru > 0) for all vectors u ^ in K m ; this is commonly expressed as r > (r > 0). 
Given r\ and r2 in S m , the inequality r\ > r-i means r\ — r^ > 0. 

(i) If r > then r" 1 > 0. 
(ii) x'x > for any matrix x. 

(iii) If r > then r is nonsingular if and only if r > 0. 
(iv) If r > 0, s > 0, and r — s > then s _1 — r _1 > and \r\ > \s\. 
(v) A symmetric matrix is positive definite (positive semi-definite) if and only 
if all its eigenvalues are positive (non-negative). 
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(vi) If r G S m then there exists an orthogonal matrix 7 e 0(m) such that 
7V7 = A where A =diag(Ai, . . . , A m ). Each Xj is real and equal to the jth eigen- 
value of r. 

(vii) If r is a positive semi-definite m x m matrix then there exists a positive 
semi-definite m x m matrix, written as r 1 ' 2 , such that r = r l ' 2 r 1 ' 2 . If r = 7A7', 
7 G O(m), then r 1 / 2 = 7 A 1 / 2 7 ' where A 1 / 2 =diag(A 1 /2 , . . . , xU, 2 )- 

11. If x is n x fc and y is m x k, n > m, then .t'x = y'y if and only if there 
exists v G V„ !m such that x = vy. In particular, if x is n x m, n > m, then there 

wy, where y = (x'x) 1 ' 2 . 
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